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FOREWORD 

This  Consulting  Report  presents  the  results  of  tests  conducted  by 
The  Human  Resources  Research  Office,  Division  No.  5 in  support  of  Joint 
Task  Force  Two  (JTF-2)  of  the  Joint  Chiefs  of  Staff. 


JTF-2  was  organized  by  the  Joint  Chiefs  of  Staff  to  conduct  a series 
of  coordinated  and  integrated  tests  to  determine  the  capabilities  and 
vulnerabilities  of  offensive  and  defensive  weapons  systems  operating 
in  the  low  altitude  regime.  Test  3- 1/3-5  Nonfiring  (NF)  provided  discrete 
operational  and  tactical  data  for  the  evlauation  of  visually  sighted  and 
radar  controlled  air  defense  weapons  systems  against  low-flying,  high 
speed  tactical  and  strategical  aircraft. 

This  report  is  concerned  with  the  results  of  that  portion  of  Test 
3. 1/3-5  (NF)  which  was  conducted  in  the  Oklahoma/Arkansas  environment 
of  Test  4.1,  Visual  Target  Acquisition. 


The  data  reduction  and  statement  analysis  for  this  test  was  provided 
to  JTF-2  by  Mr.  Michael  Carter  of  Sandia  Laboratories,  Albuquerque,  New 
Mexico.  The  support  of  the  Human  Resources  Research  Office  (HumRRO)  was 
provided  under  authority  of  the  Department  of  the  Army  in  accordance  with 
the  HumRRO  prime  contract,  DA  44-188-AR0-2 . 


ROBERT  D.  BALDWIN 
Director  of  Research 


pjffiCSDIlD  PAGE  BLAHMOT  WUOD 


UNClA^oi 


iW»qiwiy.ff«.1»wrWiW>li,Wi,itw,  ■, 


*■  •fH.n-WW'}'  *■ 


wey**1**  i***h*'«^»/ 


•KK.fYlWWl"'  ’ •■'■*  v 


svmjoc 

1.  (U)  Joint  Task  Force  Two  Test  3»l/3*5 > Surface  Based  Air  Defense  (NF) , 
was  designed  as  a series  of  coordinated  test  efforts  to  provide  opera- 
tional and  technical  data  for  the  evaluation  of  the  capabilities  of  surface 
based  air  defense  systems  against  attacks  by  low-altitude  high  performance 
aircraft. 

2.  (U)  The  Oklahcma/Arkansas  portion  of  Test  3-l/3«5  (NF)  investigated  the 
capabilities  of  ground  observers  to  detect  and  estimate  the  range  to  air- 
craft flying  attack  missions  against  prebriefed  ground  targets.  Observer 
performance  was  obtained  for  three  aircraft  flying  three  programmed  speeds 
and  two  programmed  aJ  t itudes . 

3.  (U)  The  tc-s',  environment  used  was  the  same  as  that  for  Test  4.1,  Visual 
Target  Re -ignition.  The  Test  3-l/3«5  (NF)  portion  of  the  test  was  conducted 
by  Human  Resources  Research  Office  (HumRRO) , Divison  No.  5 (Air  Defense)  on 
a cooperative  noninterference  basis.  Instrumentation  and  data  on  event 
times  as  well  as  data  on  aircraft  position,  speed,  and  altitude  were  pro- 
vided frcm  that  collected  for  Test  4.1. 

4.  (U)  During  the  conduct  of  the  test,  considerable  aircraft  position 
data  were  lost  due  to  faulty  instrumentation  and  offcourse  aircraft  flight 
paths.  During  the  data  reduction  portion,  additional  data  losses  accrued 
due  to  ambiguities  in  observer  responses  and  missing  visibility  and  unmask 
data.  As  a result,  the  original  test  objectives  for  Test  3 * l/ 3-5  (NF)  had 
to  be  considerably  modified.  Final  objectives  were  to  determine: 

a.  the  relationship  between  the  frequency  of  detection  and  the 
slant  range  to  the  aircraft, 


,MP\ 


b.  the  relationship  between  the  frequency  of  detection  and  the 
apparent  size  of  the  aircraft, 

c.  the  relative  frequency  with  which  the  aircrew  visually 
acquired  the  ground  target  before  being  detected  by  the  observers, 

d.  the  relationship  between  the  frequency  of  visual  detection 

and  the  distance  to  the  aircraft  for  each  of  the  ground  targets  separately, 
and 


e.  the  accuracy  with  which  ground  observers  judge  engagement 
ranges  of  4-00,  800,  1,500,  and  2,500  meters  under  conditions  different 
fran  those  used  in  training. 

5.  (U)  Sixteen  observers  were  used  in  the  test.  They  received  training 
in  range  estimation  at  Fort  Bliss,  Texas,  just  prior  to  the  test.  During 
the  test,  they  were  deployed  in  groups  of  4 at  each  of  the  4 ground  targets 
chosen  for  Test  3. l/ 3-5  (NF) . Observers  were  systematically  rotated  from 
target  to  target.  Observers  were  instructed  to  search  a l80-degree  sector 
for  each  trial,  with  early  warning  of  an  aircraft  approach  being  provided 
for  some  trials.  Observers  were  assigned  a range  to  estimate  for  both 
inbound  and  outbound  legs  of  the  flights.  Three  real  time  events  were 
recorded  for  each  observer  for  each  trial  by  means  of  an  observer  response 
box  (ORB)  connected  with  Test  4.1  instrumentation.  These  events  were:  time 

at  detection,  time  when  the  aircraft  was  at  the  estimated  inbound  range,  and 
time  when  the  aircraft  was  at  the  estimated  outbound  range.  In  addition  to 
the  real  time  events,  the  observer  completed  a posttrial  questionnaire  for 
each  trial.  The  questions  pertained  to  whether  or  not  smoke  was  seen  at  the 
time  of  detection,  against  what  kind  of  background  the  aircraft  was  seen, 
and  whether  or  not  the  aircraft  was  seen  before  it  was  heard. 
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6.  (U)  The  test  data  were  recorded  on  magnetic  tape  by  the  GSIP  located 
at  each  target.  These  data  were  subsequently  edited  to  remove  ambiguities 
and  time  related  to  Test  4.1  position  data.  The  resulting  responses  were 
edited  further,  and  any  data  not  considered  plausible  were  removed.  Further 
data  reduction  consisted  of  preparation  of  analysis  tapes  and  the  providing 
of  descriptive  statistics.  The  analysis  consisted  of  a review  of  the  des- 
criptive statistics  (histograms  and  scatter  diagrams)  for  the  purpose  of 
formulating  initial  analysis  of  variance  (ANOVA)  and  analysis  of  covariance 
(ANCOVA)  models.  These  models  were  processed  using  the  weighted  regression  , 
analysis  program  (WRAP)  and  multiple  analysis  of  variance  (MANOVA)  statisti- 
cal analysis  computer  programs.  The  resulting  series  of  analyses  revealed 
that  the  test  variables  were  strongly  confounded.  This  confounding  and 

the  unbalance  in  the  design  largely  precluded  any  meaningful  results  other 
than  from  descriptive  statistics.  A further  series  of  examination  revealed 
the  individual  flight  profiles  as  the  source  of  the  confounding. 

7.  (C)  The  average  distance  of  the  aircraft  at  detection  for  all  trials 
was  approximately  6,200  meters.  A cumulative  percent  detection  curve  as 

a function  of  slant  range  was  obtained  for  all  trials,  and  these  data  were 
compared  with  data  from  two  previous  detection  studies.  Considering  that 
the  three  tests  were  not  comparable  with  respect  to  aircraft,  test  environ- 
ment, or  instructions  to  observers,  the  range  of  differences  in  detection 
ranges  for  the  three  studies  were  not  considered  to  be  unusual. 

8.  (C)  Cumulative  percent  detection  as  a function  of  slant  range  was 
obtained  for  each  of  the  four  ground  targets  separately.  The  average 
slant  range  at  detection  varied  over  targets  between  5/300  and  7/700  meters. 
Differences  between  the  targets  were  partially  explainable  in  terns  of 
differences  in  gross  terrain  and  unmask  characteristics  of  the  four  targets. 

9-  (C)  A measure  of  aircraft  apparent  size  (ASA)  was  found  to  be  correlated 
with  cumulative  percent  detection.  The  regression  equation  describing  this 
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relationship  is  presented.  This  equation  is  of  particular  interest  since 
it  relates  aircraft  characteristics  and  slant  range  to  cumulative  percent 
detection. 

10.  (C)  Data  frcm  the  posttrial  questionnaire  showed  that  smoke  was  noticed 
at  the  time  of  detection  of  F-4c  aircraft  a significantly  greater  percentage 
of  the  time  than  for  the  F-105D  and  A-6A  aircraft.  A majority  of  all 
detections  (68  percent)  occurred  with  either  cloud  or  sky  background,  and 
the  majority  of  aircraft  (83  percent)  were  seen  before  they  were  heard. 

11.  (C)  Ground  observers  detected  the  aircraft  before  the  pilot  acquired 
the  ground  target  60  percent  of  the  time.  The  percentage  of  time  the 
observer  detected  first  varied  from  target  to  target.  However,  these 
differences  were  partially  explainable  in  terms  of  gross  differences  between 
target  visibilities  and  gross  differences  between  terrain  and  unmask  charac- 
teristics of  the  targets.  Also,  there  were  indications  that  the  inflight 
behavior  of  the  aircraft  had  a significant  effect  on  whether  the  observer 
detected  first  or  whether  the  pilot  acquired  first. 

12.  (C)The  range  estimation  data  indicated  that  observers  can  estimate 
the  range  of  an  outbound  aircraft  more  accurately  than  an  inbound  aircraft. 

13.  (C)  However,  the  magnitude  of  the  errors  made  during  the  test  were 
larger  than  those  made  at  the  conclusion  of  training.  It  was  hypothesized 
that  the  change  in  aircraft  type  and  test  environment  probably  contributed 
to  the  greater  errors  made  during  the  test.  Also,  the  lack  of  feedback  on 
the  accuracy  of  estimates  during  the  test  was  believed  to  be  a factor. 
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SECTION  1 
INTRODUCTION 


BACKGROUND. 


a.  Joint  Task  Force  Two  (JTF-2)  Test  3 - 1/ 3-5  Nonfiring  (NF),  Surface 
Based  Air  Defense,  was  the  third  in  a series  of  coordinated  and  integrated 
tests  designed  to  obtain  factual  operational  and  technical  data  on  low 
altitude  offensive  and  defensive  weapons  system  operation. 


b.  Tests  3*1  and  3*5  were  combined  tests.  Test  3-1  was  a test  of  the 
vulnerability  of  US  aircraft  flying  at  low  altitude  and  high  speed  aircraft 
against  Communist  gun  type  weapons.  Test  3-5  was  a test  of  the  effective- 
ness of  US  gun  type  weapons  against  low  altitude,  high  speed  Communist 
aircraft.  Test  3 - 1/3 - 5 was  originally  planned  to  be  conducted  in  two 
phases,  a nonfiring  and  a firing  phase.  The  disestablishment  of  JTF-2 
has  precluded  the  firing  stage  from  being  conducted. 


I 


c.  The  major  portion  of  the  test  was  conducted  for  JTF-2  by  the  US 
Army  Combat  Developments  Command  Experimentation  Center  (CDCEC),  Fort  Ord, 
California. 

d.  During  the  planning  of  the  nonfiring  (NF)  phase  of  Test  3»l/3«5> 
Joint  Task  Force  Two  (JTF-2)  recognized  that  data  concerning  the  ability 
of  ground  observers  to  detect  and  estimate  the  range  of  low-flying  high 
performance  aircraft  could  be  obtained  using  the  aircraft  and  instrumenta- 
tion established  for  JTF-2  Test  4.1,  Visual  Target  Acquisition.  This 
additional  data  would  augment  that  obtained  by  JTF-2  during  Test  3 * l/ 3*5 
(NF)  which  was  conducted  at  Hunter  Liggett  Military  Reservation  (HIM) , 

a subpost  of  Fort  Ord,  California. 


e.  JTF-2  Test  4.1  was  an  air-to-ground  visual  target  acquisition  test 
conducted  in  the  Oklahoma/Arkansas  area  in  the  general  vicinity  of  Mena, 
Arkansas.  One  of  the  objectives  of  Test  4.1  was  to  determine  the  abilities 
of  representative  aircrews  to  visually  acquire  prebriefed  ground  installa- 
tions consisting  of  simulated  military  targets  such  as  surface-to-air 
missile  and  radar  6ites,  landing  strips,  logistics  supply  points,  and 
bridges.  It  was  concluded  that  this  air-to-ground  target  acquisition 

test  would  provide  an  environment  for  obtaining  ground-to-air  observations, 
such  as  visual  detection  and  distance  estimation  judgments. 

f.  Test  4.1  was  conducted  under  conditions  that  offered  a unique 
environment  for  obtaining  ground-to-air  visual  observation  data  and  would 
supplement  the  very  large  amount  of  observer  data  obtained  at  HIMR. 

(1)  Although  the  aircrews  participating  in  Test  4.1  received 
normal  premission  briefings  concerning  the  location  of  the  ground  targets, 
each  aircrew  would  participate  in  only  one  flight  over  each  target. 
Consequently,  it  was  expected  that  the  crews  would  exhibit  variability  in 
their  navigational  and  target  acquisition  ability.  This  variability  in 
the  performance  of  the  aircrews  would  increase  the  uncertainty  of  the 
ground  observers  concerning  the  time  of  arrival  at  the  target  and  the 
direction  of  "attack"  by  the  aircraft.  These  conditions  of  uncertainty 
concerning  the  aircrafts'  time  of  arrival  and  direction  would  be  more 
representative  of  tactical  ground-to-air  defense  than  if  the  aircraft 
always  flew  the  same  flight  path  to  each  target. 

(2)  One  measure  of  aircrew  performance  of  interest  to  Test  4.1 
was  the  time  and  distance  at  which  the  aircrew  visually  acquired  the  ground 
target.  Since  this  pilot  acquire  event  was  recorded  for  each  trial  during 
Test  4.1,  this  event  would  be  compared  with  the  ground  observer  detection 
time  and  distance  events  to  obtain  data  relevant  to  the  following  questions: 

(a)  Does  the  air  defense  gunner  tend  to  detect  the  attacking 
aircraft  before  the  aircraft  locates  the  defended  point? 
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(b)  When  the  pilot  acquires  the  target  first,  how  much  time 
elapses  before  the  air  defense  gunner  detects  the  aircraft? 

(3)  The  instrumentation  provided  for  Test  4.1  included  distance 
measuring  equipment  (DME)  for  accurately  locating  the  position  of  the 
aircraft;  a radar  line-of-sight  system  between  aircraft  and  each  ground 
target  for  determining  the  time  at  which  the  unmask  and  remask  events 
occurred,  and  visibility  recording  equipment  at  each  target  site.  The 
DME  and  electronic  unmask  instrumentation  provided  accurate  data  concern- 
ing the  slant  distance  to  the  aircraft  from  the  target  when  the  visual 
detections  occurred  and  permitted  computation  of  unmask-to-detection  time 
delays.  Measurement  (computation)  of  the  unmask-to-detection  time  delay 
was  a potentially  important  performance  measures  that  could  be  used  to 
evaluate  the  effects  on  visual  detection  of  controlled  (independent)  test 
factors,  such  as  variation  in  predetection  altitude,  aircraft  speed,  and 
the  accuracy  of  the  early  warning  provided  to  the  ground  observers. 

2.  PREVIOUS  TESTS. 

a.  Visual  Detection.  A number  of  tests  have  been  conducted  In  desert 
terrain  concerning  human  ability  to  detect  low-flying  aircraft  under  various 
simulated  tactical  conditions. 

(l)  Tests  conducted  at  Gila  Bend,  Arizona,  by  the  US  Army  Human 
Engineering  Laboratories  (see  reference  l)  varied  the  size  of  the  sector 
to  be  searched,  but  provided  no  temporal  early  warning  (no  information  con- 
cerning probable  time-on-target  and  time  of  arrival).  When  the  search 
sector  was  4-5  degrees,  the  mean  distance  of  jet  aircraft  (T-33>  P-86,  and 
F-100)  at  detection  was  2,750  meters.  When  the  search  sector  was  90  degrees 
and  3^0  degrees,  the  average  distances  were  approximately  2,585  and  1,935 
meters,  respectively. 
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(2)  Tests  with  T-33  and  F-100  aircraft  conducted  by  White  Sands 
Missile  Range  (WSMR),  New  Mexico  (see  reference  2),  used  30-degree  and  l80- 
degree  sectors  and  varied  temporal  early  warning  from  less  than  15  minutes 
to  more  than  9°  minutes.  The  mean  detection  range  of  F-100  aircraft,  for 
example,  averaged  over  all  search  and  early  warning  conditions  was  approxi- 
mately 4,400  yards.  However,  detection  range  appeared  to  be  influenced  by 
aircraft  heading  angle,  altitude,  and  speed  well  as  other  factors.  The 
affect  of  the  lengthy  alert  intervals  on  detection  range  could  not  be 
evaluated. 

(3)  Tests  were  conducted  by  The  Human  Resources  Research  Office 
(HumRRO)  in  April  1965 , using  the  same  terrain  used  earlier  for  the  WSMR 
tests.  The  HumRRO  tests  compared  unaided  versus  b inocular-aided  detections 
using  a 30-degree  search  sector  and  1 to  5 minutes  of  early  warning.  Under 
these  conditions,  the  average  range  at  which  jet  aircraft  (F-4C,  F-100,  and 
T-33)  were  detected  was  approximately  10,000  meters,  when  averaged  over  all 
viewing  systems  and  observer  offsets.  The  increase  in  detection  range, 

as  compared  to  earlier  tests,  was  attributed  to  the  increased  accuracy  of 
the  early  warning  information  provided  the  observers  concerning  the  heading 
and  expected  1 iue  of  arrival  of  the  aircraft.  When  averaged  over  all  air- 
• craft,  the  optical  aids  did  not  reliably  increase  detection  range. 

(4)  Tests  were  also  conducted  by  HumRRO  in  June  1965/  In  conjunction 
with  the  JTF-2  Test  1.0,  Minimum  Terrain  Clearance,  at  Tonopah,  Nevada. 

Either  1 or  5 minutes  of  early  warning  was  used,  and  the  exact  heading  of 
the  aircraft  at  the  time  of  unmask  was  known.  Both  near  and  far  terrain 
masking  existed,  and  unaided  and  blnocular-aided  detection  data  were 
obtained.  The  mean  detection  range  for  F-4c  and  F-105D  aircraft,  averaged 
over  all  viewing  conditions  for  the  far  terrain  masking  condition,  exceeded 
12,000  meters.  Detection  range  was  not  increased  by  using  either  6 x 30  or 

7 x 50  binoculars,  nor  did  the  difference  between  1 versus  5 minutes  of 
temporal  early  warning  have  a reliable  influence  on  detection  range. 
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(5)  In  summary,,  the  previous  visual  detection  tests  had  been  con- 
ducted under  good  visibility  conditions  in  desert  terrain.  Most  tests  have 
occurred  under  far  terrain  masking  conditions.  The  studies  collectively 
indicated  that  increasing  the  precision  of  early  warning  information 
markedly  increased  detection  range , and  that  detection  range  was  influenced 
by  aircraft  altitude,  aspect  angle,  size,  and  speed.  However,  the  extent 
to  which  detection  ranges  experienced  in  the  desert  can  be  generalized  to 
other  terrain  and  meteorological  conditions  is  not  known.  In  addition, 
tests  of  the  effect  of  aircraft  size  (type),  speed,  and  near  terrain 
masking  on  detection  were  needed. 

b.  Range  Estimation.  Very  little  test  information  exists  concerning 
man's  ability  to  judge  when  high  speed  aircraft  are  within  the  effective 
range  of  air  defense  weapons.  In  1966,  HumRRO  conducted  several  studies 
comparing  different  methods  of  training  gunners  to  estimate  engagement 
distances  of  400,  800,  1,500,  and  2,500  meters,  the  approximate  open  and 
cease  fire  ranges  of  various  forward  area  air  defense  weapons.  Although 
the  training  did  reduce  gross  judgmental  errors  and  reduced  the  variability 
of  judgments  among  observers,  all  training  and  testing  had  been  done  in 
desert  terrain.  As  a result,  the  affect  'n  ground-to-air  distance  judgment 
of  different  aircraft  backgrounds , different  types  of  terrain,  and  aircraft 
size  were  not  known.  It  was  also  not  known  how  well  the  estimation  skills 
developed  during  the  desert  training  would  transfer  to  other  type  of  terrain 
and  aircraft. 


3.  TEST  OBJECTIVES. 

a.  As  originally  conceived,  the  tests  of  visual  detection  and  raxge 
estimation  to  be  conducted  in  the  Oklahoma/Arkansas  portion  of  Test  3 - 1/ 

3.5  (NF)  were  designed  to  evaluate  the  effects  of  specific  independent  test 
factors,  such  as  the  scheduled  variation  in  aircraft  altitudes,  speed  and 
type,  the  accuracy  of  the  early  warning  (EW)  information  which  would  be  provided 
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the  observers,  the  tine  and  distance  at  which  the  aircraft  was  unmaBked 
from  the  near  terrain  or  horizon,  and  the  ambient  visibility  and  illumina- 
tion conditions. 

b.  The  tests  that  were  designed  to  evaluate  these  factors  of  interest 
to  Test  3.1/3* 5 (NF)  were  based  on  the  following  assumptions  concerning  the 
results  of  Test  4.1. 

(1)  The  time  and  distance  at  which  each  aircraft  flight  became 
unmasked  to  each  ground  target  would  be  available  for  use  in  computing  the 
fundamental  unmask-to-detection  performance  measures  and  for  use  as  a 
covariable  in  the  evaluation  of  the  effects  of  the  test  factors  on  the 
performance  measures. 

(2)  Valid  visibility  measurements  would  be  provided  for  each  trial 
(defined  as  the  flight  of  an  aircraft  over  the  target  area).  Variation  in 
the  visibility  measurements  from  trial -to-trial  would  be  used  as  a covariable 
in  the  evaluation  of  other  test  factors.  Visibility  variation  also  would 

be  examined  for  its  effect  on  the  visual  observations. 

(3)  The  variation  that  would  exist  among  the  aircrews  in  their 
ability  to  navigate  to  each  ground  target  would  be  equated  for  each  com- 
bination of  test  variables,  such  as  the  three  aircraft  speeds  or  the  two 
programmed  flight  altitudes.  This  assumption  was  critical  because  of 
technical  requirements  of  the  planned  statistical  analysis. 

4.  TREATMENT  OF  TEST  OBJECTIVES. 

a.  The  original  objectives  established  for  the  Oklahoma/ Arkansas 
portion  of  Test  3 • l/ 3*5  (RF)  were  not  achieved  due  to  problems  associated 
with  the  instrumentation  used  for  measuring  the  aircraft  unmask  and  visi- 
bility conditions,  loss  of  ground  observer  event  data,  and  large  variations 
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in  the  altitudes  and  flight  paths  of  the  aircraft.  The  extent  and  magni- 
tude of  these  problems  was  not  ascertained  until  after  the  data  collection 
portion  had  been  completed. 

b.  During  the  data  reduction  portion  of  the  Test  4.1,  it  was  found  that 
there  were  a considerable  number  of  unaccountable  irregularities  in  the 
unmask  time  and  distance  events  which  were  based  on  radar  sensings  between 
the  aircraft  and  the  ground  target.  As  a result,  it  was  concluded  that  the 
radar  unmask  data  could  not  be  used  in  evaluating  the  results  of  either 
Test  4.1  or  Test  3 • l/ 3 - 5 (NT1)  because  the  time  at  which  the  aircraft  became 
unmasked  at  the  target  was  a significant  factor  in  the  design  of  the  air-to- 
ground  target  acquisition  tests,  terrain  surveys  were  made  and  topographical 
map  studies  were  conducted  after  data  collection  was  completed.  These 
surveys  resulted  in  the  preparation  of  a mask  profile  for  each  trial  which 
displayed  the  aircraft's  altitude  in  relation  to  the  computed  masking  due  to 
horizon  and  local  masking  features.  Because  the  exact  range  at  which  an 
aircraft  became  unmasked  for  each  trial  could  not  be  determined  with  any 
reasonable  assurance  from  the  profiles,  the  computed  unmask  ranges  asso- 
ciated with  each  ground  target  were  used  qualitatively  in  evaluating  the 
field  test  results  presented  in  this  report. 

c.  The  accuracy  of  the  visibility  measurements  made  during  each  trial 
was  similarly  questioned  by  Test  4.1.  It  was  concluded  that  these  measures 
would  not  be  used  in  any  precise  or  quantitative  analysis  of  the  field  test 
results  reported  here. 

d.  The  nonavailability  of  reliable  measurement  of  the  unmask  events 
and  visibility  conditions  made  it  necessary  to  limit  the  test  objectives. 
After  extensive  inspection  of  the  test  event  data  and  the  aircraft  flight 
profiles,  it  was  concluded  that  valid  data  was  available  for  evaluating 
the  following  test  objectives. 
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(1)  Determine  the  relationship  between  the  frequency  of  visual 
detection  and  the  slant  range  of  the  aircraft  summed  over  all  observing 
conditions . 

(2)  Determine  the  relationship  between  the  frequency  of  visual 
detection  and  the  apparent  size  of  the  aircraft  summed  over  all  observing 
conditions. 

(3)  Determine  the  relative  frequency  with  which  the  aircrew 
visually  acquired  the  ground  target  before  the  aircraft  was  detected  by 
the  observer. 

(4)  Determine  the  relative  frequency  with  which  the  ground 
observer  detected  the  aircraft  before  the  aircrew  visually  acquired  the 
ground  target. 

(5)  Describe  the  relationship  between  the  frequency  of  visual 
detection  and  the  distance  to  the  aircraft  for  each  ground  target 
separately. 

(6)  Determine  the  accuracy  with  which  ground  observers  Judge 
engagement  distances  of  400,  800,  1,500,  and  2,500  meters  under  conditions 
different  from  those  used  during  training. 
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SECTION  2 

CONDUCT  OF  THE  TEST 

1.  FIELD  TEST  OPERATIONS . 

a.  The  field,  test  was  conducted  in  a rectangular  section  of  Oklahoma/ 
Arkansas  approximately  60  NM  by  4-0  NM  centered  on  Mena,  Arkansas.  The  test 
area  consisted  of  rolling  terrain  that  provided  natural  surface  undula- 
tions suitable  for  aircraft  in  flight  to  mask  themselves  from  visual  and 
electromagnetic  line-of-sight  and  also  to  provide  visual  masking  of  certain 
target  locations.  It  was  made  up  of  forests,  meadows,  rural  roads,  and 
cultural  features  representative  of  small  to  medium  population  communities 
in  the  temperate  zone  worldwide. 


b.  The  flight  courses  to  the  targets  were  arranged  to  cross  ridge 
lines  nearly  perpendicular  to  the  final  reference  point  (FRP)  to  target 
line  to  provide  definite  unmask  events  for  each  target.  Each  aircrew  was 
oriented  to  the  geographical  location  of  the  FRP  for  each  target  and  had 
the  task  of  visually  acquiring  each  prebriefed  target. 
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c.  The  aircraft  used  for  the  3-l/3»5  portion  of  the  test  included  the 
F-4C,  A-6,  and  F-105D.  These  aircraft  flew  over  the  target  area  at  the 
speeds  and  altitudes  shown  in  Table  2-1.  A more  detailed  description  of  the 
test  area  and  aircraft  courses  is  presented  in  Report  JTF2-4.1,  Volume  2, 
"Low  Altitude  Test  4.1,  Visual  Target  Acquisition,  Field  Test  Description," 
dated  October  1967. 

2.  OBSERVERS.  Sixteen  male  enlisted  personnel,  eight  US  Army  and  eight 
US  Marine  Corps,  were  assigned  as  observers.  The  Arny  personnel  were 
selected  by  their  commanders  and  came  from  units  stationed  at  Fort  Polk, 


Fort  Sill,  and  Fort  Hood,  The  Marines  were  selected  by  a team  composed 
of  one  JTF-2  representative  and  one  Human  Resources  Research  Office  (HumRRO) 
representative.  The  Marines  were  from  the  3d  Light  AA  Missile  Battalion, 
Cherry  Point,  North  Carolina.  All  observers  had  20/20  vision, 'corrected  if 
neces'sary,  and  ranged  in  age  from  l8  to  24  years.  The  average  General 
Testing  (GT)  aptitude  score  was  111  and  ranged  from  83  to  130. 

a.  Observer  Training.  The  l6  observers,  plus  noncomissioned  officers 
(NCO),  were  trained  in  the  operation  of  test  instrumentation  equipment  and 
the  estimation  of  range  to  aircraft.  This  training  was  provided  by  HumRRO 
personnel  at  Fort  Bliss,  Texas,  approximately  one  week  before  the  field 
tests  began.  An  F-100  was  used  as  the  target  aircraft  for  the  range  estima- 
tion training.  The  aircraft  flew  38  passes  each  of  two  days.  Equal  numbers 
of  passes  were  flown  at  250  and  750  feet  altitude,  and  the  passes  were 
equally  divided  between  overhead  flights  and  2C0  meters  offset.  For  one- 
half  of  the  passes,  the  aircraft's  heading  was  north;  for  the  remainder,  the 
heading  was  south.  The  objective  of  the  training  was  to  have  each  observer 
accurately  estimate  when  the  aircraft  was  at  400,  800,  1500,  and  2500  meters 
from  the  observer's  position  on  both  the  inbound  and  outbound  portions  of 
the  pass.  A more  detailed  description  of  the  training  method  is  contained 
in  Appendix  B. 


3.  TEST  SITES. 

a.  The  observer  groups  were  located  at  four  of  the  Test  4.1  ground 
targets.  The  targets  used  were  identified  as  West  1 (Wl),  West  4 (w4) , 

East  3 (E3),  and  East  4 (E4),  located  respectively  at  Cherry  Hill,  Arkansas; 
Plunketsville,  Oklahoma;  Gravelly  and  Washita  Bridge,  Arkansas.  Aerial 
photographs  of  the  targets  are  contained  in  Report  JTF2-4.1,  Volume  2,  Field 
Test  Description,  dated  October  1967. 

b.  These  targets  were  selected  prior  to  the  test  by  a team  comprised  of 
Sandia  Laboratory  and  HumRRO  personnel  to  provide  unmask  distances  to  ground 
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targets  from  3 to  15  miles.  (See  Appendix  C for  the  observer  locations  at 
each  target. ) 

4.  GROUND  SITE  INSTRUMENTATION  PACKAGE  (GSIP) . 
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a.  General  Description. 

(1)  Each  test  site  was  equipped  with  a GSIP  provided  in  support  of 

JTF-2  Test  4.1.  The  GSIP  telemetered  the  following  information  to  recorders 
in  orbiting  C-130  aircraft:  Unmask  and  remask  time  of  the  test  aircraft, 

the  visibility  and  illumination  measurements,  and  the  occurrence  of  4 

real  time  events  and  10  nonreai  time  questions  from  each  of  the  4 observer 
response  boxes  (ORB)  at  each  site.  A complete  description  of  the  instrumenta- 
tion is  contained  in  JTF2-4.I,  Volume  2,  Field  Test  Description. 

(2)  Since  the  distance  measuring  equipment  (DME)  had  a predicted 
measurement  of  — 50  feet,  the  ground  observers  were  located  at  specified 
points  within  a circular  area  having  a 100-foot  diameter,  centered  at  the 
GSIP  box.  The  actual  observer  locations  depended  upon  site  characteristics. 
Figure  2-1  is  an  example  showing  the  arrangement  of  observer  positions  for 
Target  E3.  Each  observer  position  was  marked  with  a stake;  and  the  obser- 
vers were  geographically  oriented  to  a l80-degree  search  sector  containing 
the  expected  flight  paths  of  the  aircraft.  All  observers  at  a test  site 
were  oriented  to  the  same  l80-degree  search  sector. 

b.  Unmask  Time.  Unmask  and  remask  time  of  the  aircraft  was  to  be 
measured  by  means  of  L-band  continuous  wave  transmission  from  the  test 
aircraft  to  a receiver  at  the  GSIP. 
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c.  Aircraft  Position.  The  location  of  the  aircraft  with  reference 
to  the  ground  at  any  instant  was  determined  by  means  of  DME  carried  on 
board  orbiting  C-130  instrumentation  aircraft.  The  C-130s  received  DME 
slant  range  data  from  the  test  aircraft.  This  data,  when  combined  with 
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the  a>2  information  concerning  the  position  of  the  C-130  aircraft,  per- 
mitted computation  of  the  ground  location  and  altitude  of  the  test 
aircraft. 


d.  Timing.  All  event  timing  was  recorded  in  inter-range  instrumenta- 
tion group  (IRIG-B)  format. 


e.  Visibility  and  Illumination.  For  Test  4.1,  each  site  was  equipped 
with  instrumentation  to  .measure  the  sky/ground  ratio,  total  illumination, 
and  atmospheric  transmissivity  (scattering).  This  equipment  consisted  of 
an  illuminometcr  for  measuring  sky  illumination  and  shadow,  photometers 
for  measuring  the  sky/ground  illumination  ratio  in  the  direction  of  flight, 
and  a telephotometer  for  measuring  atmospheric  scattering. 

f.  Observer  Response  Boxes  (ORB).  Four  ORB  were  connected  by  cable  to 
the  GSIP  at  each  test  site.  The  ORB  consisted  of  two  units;  a real  time 
event  unit  worn  by  the  observer  and  a nonreal  time  question  box  placed  on 
the  ground  at  each  observer  position. 


(l)  The  real  time  section  of  the  ORB  was  operated  in  the  following 

manner: 


(a)  At  the  time  the  observer  saw  the  aircraft,  he  depressed 
the  "Detect"  button  located  at  the  top  of  the  ORB. 

(b)  When  the  observer  believed  the  aircraft  was  at  the  speci- 
fied incoming  range,  he  depressed  the  middle,  or  "inrange,"  button. 

(c)  When  the  observer  believed  the  aircraft  was  at  the  speci- 
fied outgoing  distance,  he  depressed  the  "Outrange"  button. 
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(a)  A shielded  spring-loaded  error  switch  was  located  on 
the  side  of  the  real  time  section  of  the  Gu3.  Inis  switch  was  used  by 
the  observer  to  record  any  accidental  button  pushes. 


(2)  The  nonreal  time  section  of  the  GltB  permitted  entering  the 
following  coded  information: 


J 


(a)  observer  identification, 

(b)  incoming  range  specified  for  each  observer  each  day, 

(c)  outgoing  range  specified  for  each  observer  each  day, 

(d)  early  warning  condition  fcr  each  flyover  (or  trial) 

(Yes  or  No), 

(e)  aircraft  exhaust  smoke  was  noticed  at  the  time  of  detec- 
tion (Yes  or  No), 

(f)  background  of  aircraft  at  time  of  detection  (clear  sky, 
clouds,  or  terrain),  and 

(g)  aircraft  was  heard  before  it  was  seen  (Yes  or  No) . 

5-  DAILY  SEQUENCE  OF  TEST  PROCEDURES. 

a.  Rotation  of  Targets.  The  l6  men  were  divided  at  random  into  4 
observer  groups.  A,  B,  C,  and  D,  containing  2 men  from  each  service.  The 
composition  of  the  groups  remained  constant  throughout  the  duration  of  the 
test  to  facilitate  cycling  observer  groups  to  test  sites  in  the  event  of 
aircraft  aborts,  inclement  weather,  or  other  reasons.  The  groups  were  sys- 
tematically rotated  to  all  4 test  sites  during  the  first  4 test  days.  During 
the  remaining  15  test  days,  the  groups  were  selectively  assigned  to  specific 
sites  to  provide  maximum  utilization  of  the  Test  4.1  flights. 
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b.  Test  Monitors.  There  was  one  KumRRO  monitor  located  at  each  test 
jite.  Monitors  were  rotated  periodically  between  sites  and  had  the  follow- 

CIuwlGS  • 


(l)  they  supervised  the  movement  of  observers  to  the  test  sites, 


(2)  monitors  gave  observers  the  instructions  for  the  day  and  the 
ranges  that  they  were  to  estimate, 


(3)  monitors  checked  each  CRB  to  insure  proper  connection  with  the 
GST?  box  and  to  insure  the  correct  observer  code  had  been  put  into  the  ORB, 


(4)  the  monitor,  stationed  near  the  communication  equipment, 
monitored  all  messages  from  Test  h.l  Control  and  relayed  appropriate 
information  to  the  observers,  such  as  early  warning  time,  and  aircraft 
aborts,  and 


(5)  the  monitors  recorded  the  time,  course,  test  site  number  and 
aircraft  type  for  each  trial  during  a test  day.  All  deviations  or  changes 
in  the  prescribed  test  procedure  were  recorded,  as  well  as  known  reasons 
for  missing  data. 


c.  Procedures  Prior  to  the  First  Trial.  Immediately  upon  arriving  at 
the  test  site,  the  group  monitor  called  Test  Control  and  reported  that  his 
group  was  at  their  site.  The  group  monitor  gave  the  test  instructions  to 
each  observer,  which  included  the  incoming  and  outgoing  distances  that  were 
to  be  estimated  by  each  observer.  Each  observer  at  each  site  had  1 of  l6 
combinations  of  incoming  and  outgoing  distances  each  day.  The  same  distances 
were  estimated  for  all  trials  during  the  day  for  which  complete  early  warn- 
ing was  given  by  Test  3 - 1/3 - 5 (NF)  Control.  Except  for  the  no  early  warning 
trials,  the  group  monitor  announced  the  time  the  first  trial  was  to  begin. 
Each  observer  moved  to  his  position  and  became  familiar  with  the  sector 
he  was  to  search  on  the  first  trial. 
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a.  Procedure  during  -.he  Ho  Earl y Wtmv.ing  Trials.  The  observers  were 
instructed  that  or.  certain  trials  they  would  not  be  informed  of  the  exact 
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'hey  were  instructed,  to  search  a 


sector  of  EsG  degrees.  Any  tine  the  observer  initially  detected  the  aircraft 
visually,  he  was  instructed  to  immediately  depress  the  "Detect"  button, 
even  if  the  aircraft  had  passed  his  site.  As  soon  as  the  aircraft  had  been 
visually  detected,  the  observer  answered  the  posttrial  questions.  There 
was  no  range  estimation  to  be  accomplished  on  the  no  early  warning  trials. 


e.  Procedure  for  Complete  Early  Warning  Trials, 


(1)  When  complete  early  warning  was  provided,  the  observer  knew 
within  approximately  — 20  degrees  the  expected  approach  of  the  aircraft 
and  the  approximate  time  of  arrival,  accurate  within  1 to  2 minutes.  The 
observers  soon  learned  that  most  aircraft  approached  each  test  site  from 
approximately  the  same  azimuth.  In  fact,  seme  observers  reported  that  they 
knew  the  direction  after  they  saw  one  trial  over  each  test  site. 

(2)  Test  4.1  Control  transmitted  information  to  the  test  sites 
concerning  the  aircraft's  time  of  arrival  at  designated  check  points. 

This  information  was  used  by  each  monitor  to  determine  the  probable  time 
of  arrival  of  the  aircraft  at  the  test  site.  This  early  warning  informa- 
tion was  relayed  to  the  observers. 


(3)  On  all  trials,  when  complete  early  warning  was  given,  the 
observers  also  made  two  estimates  of  range,  one  incoming  and  one  outgoing. 
After  the  aircraft  had  been  detected,  the  observers  continued  to  watch 
the  aircraft.  When  the  observer  believed  the  incoming  aircraft  was  at 
the  assigned  range  from  him,  he  was  to  depress  the  "Inrange"  button  on  his 
response  box.  He  was  to  continue  to  watch  the  aircraft  as  it  passed  over 
his  position  and  as  it  was  outbound.  When  he  believed  the  outgoing  aircraft 
was  at  the  assigned  range,  he  was  to  depress  the  "Outrange"  button  on  his 
response  box.  The  trial  was  over  when  he  made  the  outgoing  range  estima- 
tion. The  observers  then  answered  the  posttrial  questions. 
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SECTION  3 

DATA  PROCESSING  AND  ANALYSIS 

1.  GENERAL.  The  data  processing  consisted  of  reducing  and  editing  the 
Test  3. 1/3.5  (NF)  Oklahcma/Arhanaaa  event  data,  time  relating  this  event 
data  with  applicable  Test  4.1  aircraft  position  data,  and  creating  a 
data  bank  for  use  In  the  analysis  effort. 

2.  QUALITY  CONTROL  OPERATIONS. 

a.  The  event  data  vere  recorded  on  magnetic  tape  In  the  C-130 
aircraft.  These  tapes  vere  delivered  to  Sandia  Laboratory,  where  they 
vere  processed  through  a playback  station  and  formated  for  the  CDC-3400 
computer.  The  event  data  vere  processed  by  a program  called  QUICKORB, 
which  provided  preliminary  Information  which  was  transmitted  to  the  Mena 
Test  Range  concerning  the  availability  of  event  data  from  completed 
trials. 

b.  The  data  event  tapes  vere  edited  manually  to  remove  Invalid 
trials  and  individual  observations. 

(1)  AUevent  data  associated  with  a trial  (that  is,  the  flight 
of  an  aircraft  near  a ground  target)  vere  deleted  if  either  the  aircraft 
was  grossly  off-course,  or  it  penetrated  the  target  area  outside  the 
assigned  l80-degree  search  sector,  or  no  aircraft  position  data  vere 
available. 

(2)  An  individual  observation  was  deleted  if  the  event  occurred 
out  of  sequence  or  at  an  improper  time  during  a trial:  For  example.  If 
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a detect  event  was  recorded  for  an  observer  after  he  made  an  Inrange 
estimate,  the  detect  event  vas  discarded. 

3.  DAIA  REDUCTION. 

a.  The  ground  observer  event  data  were  time-related  to  Test  4.1 
aircraft  position  data  by  use  of  computer  programs.  At  this  time,  the 
performance  measures  were  computed,  as  were  the  values  of  the  independent 
variables  to  be  used  in  subsequent  statistical  analyses. 

b.  The  Independent  variables  calculated  for  each  event  Included 
such  factors  as  the  aircraft  slant  range  (SR),  the  sun  angle  (SA),  and 
aircraft  angular  velocity. 

c.  The  performance  measures  Included  detection  distance,  algebraic 
range  estimation  errors,  and  the  pilot  acquire  minus  observer  detection 
time  interval. 

4.  ANALYSIS  PIAN.  This  section  describes  the  general  approach  used 
for  statistical  analyses  of  the  data.  A more  detailed  discussion  of 
the  analysis  approach  and  methods  is  presented  in  Appendix  D. 

a.  Descriptive  statistics  included  histograms  and  cumulative 
frequency  plots  vhich  presented  the  frequency  of  occurrence  of  the 
specific  values  of  the  performance  measures  and  Independent  variables; 
and  scatter  diagrams  (SCD),  vhich  graphically  presented  the  concurrent 
frequency  distributions  of  tvo  measures,  such  as  the  concurrent  (or  x,  y) 
distributions  of  detection  range  and  aircraft  angular  velocity.  Also 
computed  for  each  variable  were  the  mean,  median,  standard  deviation  and 
maximum  and  minimum  values  used. 


b.  Analysis  of  variance  (ANOVA) , analysis  of  covariance  (ANCOVA) , 
and  regression  analysis  (RA)  methods  were  subsequently  used  to  evaluate 
the  statistical  reliability  of  the  data  and  the  strength  of  the  dependency 
relationships  among  the  Independent  variables  and  the  performance  measures. 
A technical  discussion  of  the  scope  of  these  analyses  also  appears  In 
Appendix  D. 

5.  EFFECT  OP  DATA  REDUCTION  ON  THE  ANALYSIS  PIAN. 

a.  As  discussed  In  Section  1,  a number  of  problems  with  Test  4.1 
Instrumentation  became  evident  during  the  data  reduction  phase  of  that 
test.  Data  losses,  absence  of  unmask  and  visibility  data,  and  the 
aircrews'  frequent  failure  to  fly  assigned  altitudes  and  speeds  made 
It  necessary  to  alter  the  original  analysis  plan. 


b.  Following  the  data  reduction  phase,  it  was  known  that  the 
numbers  of  observations  actually  obtained  for  each  combination  of 
Independent  variables  were  very  unbalanced.  Table  3-1  presents  the 
detection  data  matrix  that  was  available  following  the  data  reduction 
phase  for  each  combination  of  observer,  ground  target,  aircraft,  and 
early  warning  level.  The  total  lack  of  observations  in  many  of  the 
cells  of  Table  3-1  indicates  the  extent  of  the  unbalance. 

c.  Although  valid  analysis  procedures  were  performed  on  the  data, 
the  results  were  uninterpretable.  The  inability  to  interpret  the  results 
of  the  ANOVA  and  RA  was  primarily  attributed  to  the  trial-to-trial  varia- 
tion in  the  aircrafts'  flight  paths,  speeds,  and  altitudes.  These  sources 
of  unequal  variation  were  confounded  with  the  lack  of  balance  in  the  number 
of  observations  obtained  for  each  cell  of  the  data  matrix. 

d.  If  unmask  and  visibility  measures  had  been  available,  the  trial- 
to-trial  variation  in  detection  ranges  as  a function  of  the  aircrafts' 
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flight  characteristics  could  have  been  adjusted  by  statistical  correlation 
techniques.  In  the  absence  of  unmask  data,  the  variations  in  detection 
range  could  not  be  meaningfully  attributed  to  any  of  the  independent 
variables.  Similar  confounding  of  uncontrolled  variables  with  test 
variables  affected  the  analyses  of  the  distance  estimation  errors.  The 
net  result  vas  a decision  to  limit  the  analyses  to  the  gross  descriptive 
statistics  for  each  of  the  performance  measures. 
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SECTION  U 

RESULTS  AND  DISCUSSION 


1 . (U)GTRO DUCT ION.  This  section  presents  the  results  and  discussion  of  the 
data  pertaining  to  aircraft  detection,  the  pcsttrial  questionnaire,  A (tine 
differences  between  aircrew  acquisition  and  observer  detection),  and  range 
estimation.  The  major  emphasis  in  the  presentations  is  on  descriptive 
statistics.  Results  of  the  more  sophisticated  analyses  are  not  presented 
for  the  reasons  already  cited  in  Section  3* 


2.  (CpETECTIGr 


a.  The  functional  relationship  between  cumulative  percent  detection 
and  aircraft  slant  range  was  determined  for  the  6 j3  observations  available 
in  the  data  bank.  These  data  indicate  that  the  aircraft  were  detected 
50  percent  of  the  time  before  they  were  approximately  5000  meters  from  the 
ground  target.  Ninety  percent  of  the  detections  occurred  at  a range  of 
3000  meters  or  greater,  and  ten  percent  of  the  aircraft  were  detected  at 
12,500  meters  or  greater.  The  relationship  between  cumulative  percent 
detection  and  slant  range  is  presented  in  Figure  4-1  and  may  be 


approximated  by  C = 3-639  -.37098  Ln  R,  where  C = cumulative 
P P 


percent  detection,  and  R = slant  range  in  meters. 


b.  Figure  4-1  presents  a comparison  of  the  overall  detection  performance 
obtained  in  the  Oklahoma/Arkansas  Test  and  the  results  of  previous  detec- 
tion tests  reported  by  the  Human  Engineering  Laboratories  (see  reference  l) 
and  HumRRO  (see  reference  5)  which  were  conducted  in  a desert  environment. 

The  HumRRO  test  used  a search  sector  of  less  than  30  degrees,  and  temporal 
early  warning  was  provided  within  5 minutes  of  the  trial.  The  Human 
Engineering  Laboratories'  (HEL)  test  used  search  sectors  up  to  360  degrees 
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Figure  4-1  Percent  Detection  Compared  with  Prior  Field  Studies.  (U) 


and  no  temporal  early  warning.  The  Teat  5 - -/5 - 5 (is?)  Oklahoma/Arkansas 
detection  data  is  contained  within  the  envelope  described  by  these 
earlier  tests.  Further  comparisons  are  tenuous  since  different  aircraft 
are  represented  in  the  tare  - v--.  vs.  The  ITumRFO  test  used  F-4,  F-lOO,  and 
T-33  aircraft.  The  KSL  tes.  . .ud  F-1C0,  T-33>  and  F-86  aircraft.  The 
different  results  obtained  , *,ce  three  tests  may  be,  in  part,  due  to 
differences  in  terrain  environment,  search  sector  used,  early  warning 
conditions  and  aircraft  characteristics.  The  KumRRO  test  may  represent 
the  ideal  field  detection  situation  because  of  the  narrow  search  sector, 
imminent  early  warning,  desert  visibility  and  unobstructed  terrain  used. 
The  HEL  test  may  represent  the  worst  euou  detection  situation  where  excel- 
lent terrain  and  meteorological  conditions  are  employed  because  of  the 
large  search  sectors  used  and  luck  of  temporal  early  warning.  The  Test 
3 • l/3 *5  («F)  detection  data  represent  a more  typical  field  detection 
situation  where  warned  observers  are  deployed  at  tactical  ground  targets, 
and  the  terrain  and  meteorological  environments  were  representative  of 
tactical  conditions. 

c.  Percent  detection  as  a function  of  slant  range  does  not  reflect 
aircraft  characteristics.  A measure  of  detection  performance  which 
reflects  variation  in  aircraft  size  and  heading  is  the  aircraft  subtended 
angle  (ASA).  ASA  is  the  angle  subtended  by  the  diameter  of  a circle 
having  an  area  equal  to  that  presented  by  an  aircraft  at  a specified 
slant  range  from  the  observer. 


d.  Cumulative  percent  detections  is  correlated  .965  with  ASA.  This 
relationship  can  be  used  to  predict  the  probability  of  detection  given  an 
aircraft  slant  range.  The  equation  describing  this  relationship  is: 

= 2.717  + .316  Ln  ASA,  where  Cp  = cumulative  probability  of  detection, 
and  ASA  = aircraft  subtended  angle  in  radians.  ASA  may  be  approximated  as 


target  area 


ASA  = 2 x tan 
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e.  The  area  preserved  by  the  target  used  in  the  ASA  equation  may  be  approx 


imated 


as  Target  Area  - 


A.,  x R.  + A„  xH  +■  A_. 

j C z v 

R 


where 


Ag  = aircraft  side  area,  = aircraft  bottom  area 
area,  RQ  = flight  path  offset  from  observer,  K = aircraft  altitude,  and 
R = aircraft  slant  range  of  interest. 


, Ay  = aircraft  frontal 


f.  Figure  1-2  presents  the  cumulative  percent  detection  as  a function 
of  aircraft  slant  range  for  each  of  the  four  ground  targets  used  in  the 
test.  A quantitative  description  of  the  target  sites  is  not  available; 
however,  ground  survey  data  was  available  which  described  the  computed 
masking  conditions  at  the  target  sites.  Figure  1-3  presents  the  computed 
unmask  profiles  for  each  site.  These  profiles  indicate  the  altitude  at 
which  an  aircraft  had  to  fly  in  order  to  be  visually  unmasked  for  each 
ground  target  as  a function  of  ground  range.  These  profiles  assume  that 
the  aircraft  flew  on  course,  that  slant  range  and  ground  range  were 
equivalent,  and  that  no  near  mask  such  as  trees  obscured  the  observer's 
vision. 


g.  Inspection  of  these  profiles  indicated  that  targets  Wl’  and  Wl  had 
very  similar  unmask  profiles.  As  might  be  expected  under  this  circumstance, 
the  cumulative  percent  detection  plots  for  targets  W1  and  Wl  are  very 
similar.  The  cumulative  detection  functions  for  targets  E3  and  El  appear 
quite  different  both  from  each  other  and  from  targets  Wl  and  Wl.  On  target 
E3,  the  abrupt  change  in  mask  altitude  at  a ground  range  of  65OO  meters 
(Figure  1-3)  represents  a ridge  line.  Inspection  of  topological  maps 
indicated  that  beyond  this  ridge  line,  aircraft  cculd  fly  below  the  mask 
altitude  and  would  be  visually  masked.  It  may  be  inferred  from  the  cumula- 
tive percent  detections  (Figure  1-2)  at  target  E3  that  the  aircraft  were, 
in  fact,  rarely  available  for  visual  defection  prior  to  crossing  the  ridge 
line. 
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Figure  4-?  Computed  Unmask  Profiles  for  Each  Ground  Target  (u) 


wnn"" 


'WWliW 


I 

I 

I 

I 

I 

I 


h.  The  increase  in  masking  altitude  at  24  apparently  was  due 

to  a gradual  rise  in  the  terrain  heigr.u.  Inspection  of  topological  maps 
indicated  that  aircraft  :’_yi..j  xae  trie  Ted  coarse  would  not  fly  below  the 
masx  altitude  at  thin  target.  T..e  cumulative  percent  detections  for  this 
ground  target  reflect  the  influence  of  this  terrain  characteristics  in 
that  the  largest  proportion  of  detections  in  the  5,000  to  15,000  meter 
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3.  (C)  POSTTRIAL  QUESTIONS: 
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a.  Scope.  After  each  trial,  the  Orour.d  observers  were  required  to 
answer  a series  of  questions  concerning  aircraft  and  trial  conditions. 
Three  questions  of  potential  significance  for  detection  concerned  the 
aircraft's  background  at  the  cine  it  was  detected,  whether  or  not  the 
exhaust  smoke  was  seen  rather  than  the  aircraft  itself,  and  was  it 
heard  before  it  was  seen. 
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b.  Questionnaire  Results.  Table  4-1  presents  the  frequency  with 
which  the  observers  reported  detecting  the  exhaust  smoke  and  the  reported 
background  of  each  type  of  aircraft  at  the  time  of  detection. 

c.  Aircraft  Background.  For  53  percent  of  the  observations  (439  of 
749)  the  aircraft  were  reported  to  be  against  a cloudy  background  when 
detected.  In  32  percent  (238  of  7^9)  of  the  cases,  the  aircraft  were 
reported  as  viewed  against  a terrain  background.  In  the  remainder  of  the 
observations  the  observers  reported  that  the  aircraft  -were  against  a clear 
sky  background. 

d.  Smoke. 


(l)  The  relative  frequency  with  which  the  observers  reported 
detecting  the  aircraft  because  of  exhaust  smoke  was  largest  (70  percent) 


r'-4v. 


— 1 c ^ v for 


u'v>a  -.0*  o.V*^xVc 


.0 


v.wre 


.n.untages  of 
.'cxi  ..o  ;ely  equal. 


0 

D 

0 

B 

0 


1 

! 

! 

I 

I 

I 


^ y n*»C  V»i.  ix  wi*  av*  kj x - 1 Cx  VcL I*  x.  fcl.  /*  C *.  ^ J -...XL*!  [I^vs  0 “ A ciy 

*tvc uiCu  ex.  j s/ ..y  dxffciv..jj  airerait 

i*c  —a.’w i Vc  ircCUc^cy  Ca.  exhuuj <d.c ‘C6CIaw'4  c..  o£i£  ?—t+C  was 

y ^ — c. dr w X*  a. W—  w**w  ».' “* 5*^  cti*4A  xa. “■-'xx _j  * .ewe ve ^ w* .o  "t w o an*- 

C^u.1  L ^*0.  IaOw  dine*  jTC  ...  av*.  *»>  A.y  -_*i  "wi*.  a 3 **«33pc  w'J  ■ 


c » yl.x‘Uc;.A.  Vc**SU3  auuxwO^'/  a^c'uc.  0*010.1*  tu^H  ex  v'cl'ci^ed.  .»*,«.  1*  0......  c**icij.3; 

C*  V*  («•.  •>  VvOi  C * C pOr  ^ k-  Va  u J i.C*  V W %J  w vj  il  l.CCil^  Va  JC  a Ua  ^ ^6  A r<lQ  V A kl  -*■>-*»  - - j AW  V JC  UCUi 

i or  g.^aj"  f pci*ce*e't/  ^ a w. .u  . . ...ca'o  m*oi*o  nee  s \n  i i c a — iyc  ee  eo 

e Vc*aUX  w 0 U,  aX  a C a 0 n Cc^  U«.4V..0  (*aa  O a w.  w vI*  U..w  A »•-»-_  \ A.  <a.  w*aXA.  "t  CL  a ^fVj  wJ  . 


Me). 


. xiA>- 


,:iio..  r_.D 


UaA/a^XWa 


cl  • 1. 1 oi  "w>e  wtC  c/ j.  on  • 

(i)  one  of  one  p_rf... .\..a..oe  measures  selected  for  the  Oklnhc.rx/ 
Arkansas  pert.: on  of  Test  s.l's-l  (kf)  was  the  difference  between  „he  time 
the  aircrew  visually  acquire  j.  the  ground  target  and  the  time  the  grc;u:d 
observer  visually  detected  tv.e  aircraft.  This  difference,  va.,  computed 
in  hundreths  of  a second  for  each  observer  detect  event  for  each  target  air- 
craft encounter.  The  quant. ty  is  positive  for  those  events  in  which  the 
observer  detected  the  aircraft  first,  and  negative  for  those  events  In  which 
the  aircrew  acquired  the  t .rjet  first. 


(2}  There  is  a presumed  advantage  in  locating  the  enemy  f.  rst.  If 
the  grou..d  observer  det-.r  s the  aircraft  first,  the  defenses  can  x..  made 
ready,  and  it  may  be  p..„.oJcle  to  launch  an  attack  before  the  aircraft  has 
opportunity  to  take  evasive  action.  On  the  c -her  hand,  if  the  aircrew 
acquires  the  ground  tar.,:t  before  the  ground  observer  detects  thu  aircraft, 
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the  aircraft  has  the  advantage  in  that  evasive  action  can  be  taken,  and 
preparations  can  be  made  ior  the  type  oi  ground  defenses  expected.  For 
these  reasons,  information  on  who  detects  first,  and  by  how  long  a time, 
was  considered  important. 

b.  Performance  Cver  All  Trials.  Figure  4-4  is  a histogram  showing 
the  distribution  of  across  ala  trials.  The  overall  range  of  ^ is 
quite  large,  ranging  from  - 75*45  seconds  to  79*45  seconds.  The  mean 
of  + 6.51  seconds  indicates  that,  on  the  average,  the  observer  detected 
before  the  aircrew  acquired.  Approximately  28  percent  of  the  ^ are 
within  the  range  of  — 5 seconds. 

c.  Differences  Among  Targets. 


(l)  Table  4-2  presents  the  mean  acquisition  and  detecting  ranges 
and  ^ data  for  each  of  the  ground  targets.  The  differences  obtained 
between  the  targets  are  not  difficult  to  explain  in  terms  of  gross  charac- 
teristics of  the  targets.  As  shown  in  Figure  4-3,  unmask  conditions  for 
targets  Wi  and  W4  were  approximately  the  same.  However,  W1  was  a radar  site 
consisting  of  vehicles  and  equipment  painted  olive  drab  located  on  relatively 
low  ground,  while  W4  was  a SAM  site,  containing  missiles  painted  white,  which 
was  located  on  the  side  of  a hill.  The  observers  detected  the  aircraft  at 
the  radar  site  before  pilot  acquisition  far  more  frequently  than  they  did 
at  the  SAM  site  containing  white  missiles.  Since  the  aircraft  unmask  con- 
ditions were  similar,  the  mean  detection  ranges  for  these  two  sites  differ 
very  little,  while  the  mean  acquisition  range  at  the  radar  site  was  slightly 
less  than  half  of  that  for  the  SAM  site.  Although  the  observers  detected 
first  only  50  percent  of  the  time  at  W4,  the  mean  detection  range  was  some 
600  meters  larger  than  the  mean  acquisition  range.  This  difference  was  due 
to  the  fact  that  there  were  a number  of  detections  at  very  long  ranges  (over 
20,000  meters)  at  this  target,  with  no  aircrew  acquisitions  occurring  at 
comparable  ranges. 
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(2)  At  target  E3  the  acquisition  and  detection  ranges  were 
approximately  the  same.  This  target  was  an  airstrip  so  located  that  unmask 
was  normally  relatively  short,  occurring  as  the  aircraft  flew  over  a ridge 
line  approximately  6,500  meters  from  the  target.  At  this  unmask  range, 
aircraft  should  he  plainly  visible.  Also,  at  this  range,  a target  as 
large  as  an  airstrip  should  have  been  plainly  visible  from  the  air.  There- 
fore, it  is  not  surprising  that  pilot  acquisition  and  observer  detection 
occurred  at  approximately  the  same  time  and  same  range.  While  the  observers 
tended  to  detect  before  acquisition  somewhat  over  half  the  time,  neither 
the  difference  between  the  acquisition  and  detection  ranges  nor  the  mean 

^ for  the  target  indicate  any  advantage  for  either  aircrew  or  observer. 

(3)  Target  E4  was  a bridge,  and  unmask  range  was  quite  large, 
although  rernask  was  easily  possible.  Both  aircrew  acquisition  and  obser- 
ver detection  ranges  were  larger  for  this  target  than  any  other.  The 
average  ^ data  indicated  sane  advantage  for  the  ground  observer. 

d.  Trial  Effects. 

(l)  For  any  given  trial,  either  none,  one,  two,  three  or  all  four 
observers  stationed  at  a target  could  detect  before  the  aircrew  acquired. 

Table  4-3  presents  the  percentages  each  of  these  events  occurred  during 
the  134  trials  where  responses  were  available  for  all  four  observers.  For 
42  percent  of  the  trials,  all.  4 observers  detected  prior  to  aircrew  acquisi- 
tion, and  for  23  percent  of  the  trials  the  aircrew  acquired  the  target  before 
any  of  the  observers  detected  the  aircraft.  The  figures  in  the  row  labeled 
Expected  Percentage  were  obtained  from  the  Binomial  Theorem  by  assuming  that 
the  probability  any  given  observer  would  detect  before  aircrew  acquisition  on 
any  trial  was  0.60  (that  is,  the  same  as  the  proportion  of  positive  ^ that 
occurred) . The  analysis  indicated  that  either  none  of  the  observers  or  all 
four  of  the  observers  detected  first  far  more  frequently  than  would  be 
expected.  (This  indicates  that  the  observations  were  not  independent, 
as  assumed.)  Although  this  result  may  be  due  to  differences  in  the  manner  in 
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Table  4-3  Percent  of  Observers  Who  Detected  the  Aircraft  Before  Pilot 
Acquisition  for  Trials  Where  All  Four  Observer  Responses 
Were  Available  (U) 
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which  ;rcraft  were  flown  on  different  trials,  flight  profile  data 

were  not,  available  to  examine  this  post  facto  hypothesis. 

(2)  A further  observation  can  be  made  from  Table  4-3*  It  was 
noted  from  Table  4-2  that  the  overall  likelihood  of  a single  observer 
detecting  before  aircraft  acquisition  was  60  percent.  However,  the  data 
in  Table  4-3  indicate  that  at  least  one  of  a crew  of  four  observers 
making  independent  judgments  would  detect  prior  to  aircrew  acquisition  77 
percent  of  the  time.  This  result  suggests  that  utilization  of  multiple 
ground  observers  would  provide  a definite  engagement  advantage  to  ground 
based  air  defense  weapons. 

e.  Discussion.  It  is  assumed  that  a distinct  tactical  advantage  is 
gained  by  a ground  defense  if  ground  observers  detect  an  attacking  air- 
craft before  the  crew  of  the  aircraft  acquire  the  ground  target.  The 
data  available  indicate  that  the  ground  observer  does  have  some  overall 
advantage.  Individually,  ground  observers  detected  the  aircraft  prior  to 
aircrew  acquisition  60  percent  of  the  time.  However,  if  the  observers 
were  considered  to  be  working  as  teams  of  four  men,  at  least  one  member 
of  the  team  would  have  detected  the  aircraft  before  aircrew  acquisition 
77  percent  of  the  time.  There  are  indications  that  the  visibility  of  the 
target  from  the  air  and  unmask  range  both  affect  the  likelihood  of  the 
observer  detecting  first.  Also,  there  are  indications  that  the  behavior 
of  the  aircraft  in  flight  has  a significant  effect  on  the  likelihood  that 
either  detection  or  acquisition  will  occur  first. 

5.  (C)  POSTTEST  INTERVIEWS. 

Individual  interviews  were  conducted  with  each  of  the  observers 
during  the  last  three  days  of  testing.  The  12  questions  and  the  observers' 
responses  are  presented  in  Appendix  B.  For  the  majority  of  the  questions 
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no  cleu  nsensus  was  reflected  In  the  observers'  responses.  The  obser- 
vers din  agree  on  the  following  points: 


(l)  smoke  output  caused  the  F-4C  to  be  the  easiest  aircraft  to 


detect , 


(2)  each  observer  tended  to  return  to  his  "favorite"  lookout 
point  at  each  of  the  targets, 

(3)  early  warning  information  was  heard  via  the  radio  located  at 
the  target  area  during  the  no  early  warning  test  trials.  This  information 

was  useful  for  target  Wl,  but  of  little  assistance  on  the  other  three  targets,  and 


(4)  the  observers  also  reported  that  they  had  a pretty  accurate  idea 
of  where  the  target  would  appear  by  the  end  of  the  first  test  week.  This  con- 
tention is  supported  by  the  size  of  the  search  areas  reported  for  the  four 
targets.  On  the  average,  the  search  sectors  were  reported  to  vary  between 
approximately  50  and  90  degrees  even  though  the  observers  had  been  instructed 
to  search  a l80-degree  sector. 

6.  (C) RANGE  ESTIMATION  ACCURACY. 


a.  The  range  estimation  results  are  presented  graphically  in  Figures 
4-5  through  4-12  and  are  summarized  in  Table  4-4.  These  figures  and  table 
describe  range  estimation  performance  under  the  following  conditions: 


(1)  test  conditions  (before  training,  after  training,  first  week 
field  test,  and  last  week  field  test), 

(2)  direction  of  aircraft  flight  (incoming  or  outgoing),  and 


(3)  engagement  range  to  be  estimated  (400,  800,  1,500*  and  2,500 


meters) . 


to 

B 

0,  I 

O -S 

■ H 

1 

o Ui 

go 
O q 
< ^ 

-*  Qc 

5 * 

* O 


-IT 


250  A 500  750  1000  1250 


£*6)m 


Assigned  Range  800  Incoming 


i 


>JL 


/ w 


olrUD 


i 


i 


'Cl  ASSIF1ED 


igure  k-'J  (U) 
b-20 


LAssigned  Range  800  Outgoing 


Assigned  Range  1500  Incoming 


Assigned  Range  2500  Outgoing 


Course 

Assigned 

Hange 

400 

Incoming 

800 

1500 

2500 

Outgoing 


Deviation 


X 


Before 

Training 

After 

Training 

797 

357 

162 

73 

1394 

789 

259 

120 

1752 

1307 

233 

188 

2790 

2209 

282 

212 

711 

37  6 

15^ 

55 

764 

642 

84 

77 

1514 

il3fc 

123 

94 

2317 

2017 

203 

128 

First  Week  Last  Week 


61*8 


213 


1093 


1(85 


11(82 


672 


2428 


941 


459 


294 


748 


171 


IO85 


197 


2797 


328 


830 


352 


1335 


420 


1692 


408 


3500 


909 


498 


306 


472 


138 


1644 


440 


3088 


446 


1 

, 

t 

• 

■* 

. 

> 

\tO 


b.  An  inspection  of  the  before  and  after  training  data  indicates 
that,  in  general,  training  served  to  decrease  estimation  error.  Negative 
training  effects,  which  were  partly  caused  by  an  instrumentation  bias  in 
the  training  procedure,  were  obtained  for  some  of  the  longer  ranges  on 
the  outgoing  portion  of  the  flight  path.  The  training  was  most  effective 
in  reducing  trial-to-trial  variability  of  estimation  errors,  as  shown  by 

a reduction  in  the  size  of  the  standard  deviation  for  each  assigned  range. 
In  general,  the  training  levels  achieved  by  these  observers  were  comparable 
to  the  levels  of  performance  obtained  during  previous  range  estimation 
training  investigations  (see  reference  6). 

c.  One  objective  of  the  range  estimation  analysis  was  to  compare 
proficiency  during  training,  which  was  given  under  one  set  of  conditions, 
with  the  test  performance  of  the  trained  observers  obtained  under  a com- 
pletely different  set  of  conditions . The  training  was  accomplished  in  a 
desert  environment  under  excellent  visibility  conditions  with  one  aircraft 
which  flew  a constant  speed  at  two  programmed  altitudes  and  offsets.  The 
testing,  however,  took  place  in  a semi -mountainous  region  with  very  high 
humidity,  which  tended  to  reduce  visibility.  The  test  environment  also 
included  three  different  aircraft  which  flew  numerous  speeds,  altitudes, 
and  offsets  over  four  different  test  sites. 

d.  It  was  expected  that  the  influence  of  changing  environmental  and 
stimulus  conditions  from  training  to  testing  could  best  be  determined  by 
comparing  the  end-of -training  scores  with  the  scores  for  the  first  week 
of  the  test. 

e.  Figures  4-5  through  4-12  indicate  that  estimation  errors  increased 
from  the  training  condition  to  the  test  condition  for  the  1500-meter  and 
2500-meter  incoming  estimates  and  the  800-meter  and  2500-meter  outgoing 
estimates.  In  general,  the  average  estimation  errors  slightly  increased 
from  the  training  to  the  test  environment,  but  the  variability  of  the 
estimation  errors  drastically  increased  between  the  after  training  (T) 
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and  the  field  test  first  week  (F^)  testing  conditions.  Voss  and  Wickens 
(see  reference  7)  found  that  when  observers  were  trained  to  estimate  a 
range  of  1,700  yards  under  one  set  of  conditions,  the  accuracy  and  variability 
of  estimation  remained  approximately  the  same  over  a 6-day  period.  Since 
there  was  a 5-day  period  between  end-of -training  and  testing  for  the 
Oklahoma/Arkansas  test,  the  complete  change  of  environmental  and  stimulus 
conditions  appears  to  be  at  least  partly  responsible  for  the  large  increase 
in  variability  of  estimation. 

f.  At  the  end  of  the  fifth  week  in  the  test  environment,  the  overall 
variability  of  estimation  remained  approximately  as  it  was  after  the  first 
week,  but  the  accuracy  of  estimation  was  decreased.  This  finding  is 
inconsistent  with  results  obtained  by  Horowitz  and  Kappauf  (see  reference  8). 
They  found  that  range  estimation  performance  after  training  was  stable  for  a 
period  of  60  days  without  additional  training. 

g.  The  most  consistent  result  of  the  range  estimation  evaluation 
was  the  occurrence  of  large  variability  in  the  estimation  errors  during 
the  Oklahoma/Arkansas  testing.  As  shown  in  Table  4-4,  the  standard  devia- 
tions of  the  errors  were  very  large.  These  results  suggest  that  retention 
of  this  skill  deteriorates  rapidly  over  time,  particularly  when  no  feedback 
concerning  error  magnitude  is  available  to  the  observers,  and  the  environ- 
ment is  much  different  from  that  used  in  training. 


h.  HumRRO  has  reported  a series  of  studies  concerning  range  estima- 
tion accuracy  (see  reference  6),  which  included  a comparison  of  the 
accuracy  of  judging  a 350-meter  distance  with  and  without  the  use  of  an 
occluding  or  stadimetric  aid. 

i.  In  one  of  the  HumRRO  studies,  men  were  trained  to  estimate  350 
meters  distance  to  an  aircraft.  One  group  of  men  were  trained  using 
techniques  similar  to  those  subsequently  used  for  the  3-1/3-5  observers. 

' 4-27 


J.A'i w.  i.  k ■af’jPl.Ji*.  UCt  .-t.Ur.aKl>.  ' 


i 


\ 

\ 


A second  group  of  men  were  trained  to  use  their  index  finger,  which  occludes 
a liaison  aircraft  at  approximately  350  meters,  as  a stadimetric  aid.  Use 
of  the  occluding  device  was  found  to  reduce  average  bias  and  variability 
in  comparison  with  the  unaided  training.  It  was  incidentally  learned  that 
the  front  sight  guards,  or  tangs,  of  military  rifles  also  could  serve  as 
the  job  aid  for  determining  when  to  open  and  cease  fire  against  aircraft. 

i.  In  1968,  HumRRO  began  a study  to  identify  existing  components  or 
appendages  on  US  air  defense  weapons  which  would  function  as  stadimetric 
aids.  The  results  of  the  3 • l/3 - 5 distance  estimation  tests  support  the 
need  for  some  type  of  simple  job  aid  which  gunners  could  use  to  estimate 
the  open  and  cease  fire  events. 
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APPENDIX  B 

RANGE  ESTIMATION  TRAINING 


1.  INTRODUCTION. 


a.  Previous  field  studies  conducted  at  Fort  Bliss,  Texas,  concerning 
range  estimation  training  methods  indicated  that  instruction  using  immediate 
knowledge  of  results  is  the  most  effective  and  efficient  method.  The  use  of 
this  method  had  resulted  in  rapid  improvement  in  a short  period  of  time  with 
smaller  errors  than  when  other  methods  were  used.  For  these  reasons,  the 
method  of  immediate  knowledge  of  results  was  selected  for  training  the  16 
ch  servers. 


b.  The  training  was  conducted  over  a three-day  period.  At  the 
end  of  each  day's  training  session,  a test  was  given  to  determine  each 
individual's  status  as  training  progressed.  A test  was  also  given  before 
the  first  training  session  to  provide  a performance  baseline  in  order  to 
evaluate  the  effects  of  the  training. 


2.  DESCRIPTION  OF  TRAINING  ENVIRONMENT. 


a.  The  training  was  conducted  during  late  morning  and  early 
afternoon  hours  at  Hueco  Range  No.  2,  Fort  Bliss,  Texas.  The  relatively 
flat  desert  terrain  provided  for  meteorological  range  of  approximately  75 
miles.  To  the  near  west  and  distant  north  there  was  a mountainous  back- 
ground, and  northeaut,  east,  and  south  there  was  sky  background. 


b.  In  order  to  reduce  the  possible  influence  of  terrain  features 
as  cues  for  range  estimation,  three  training  sites,  several  thousand 
meters  apart,  were  used.  Training  was  conducted  at  a different  site  each 
day. 
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c.  Two  parallel  flight  paths  were  set  up  for  the  F-100  target 
aircraft  to  follow.  A "red"  course  passed  200  meters  to  the  west  of  each 
site,  and  a "yellow"  course  passed  directly  through  each  site.  The  air- 
craft flew  at  one  speed,  400  knots  true  air  speed,  but  used  two  altitudes 
in  order  to  vary  the  aircraft  aspect.  These  altitudes  were  250  feet  and 
75°  feet.  Alternately  the  aircraft  flew  from  the  north  and  south. 

3.  TRAINING  PROCEDURE. 

a.  The  observers  were  initially  instructed  as  to  the  nature  of 
the  training.  They  were  then  positioned  so  that  each  could  see  the  entire 
flight  path  in  both  directions.  The  instructor  prepared  the  observers  with 
a warning  (READ!)  a few  seconds  before  each  signal  to  estimate  was  given. 

As  the  aircraft  flew  over  the  course,  the  observers  made  two  estimates  of 
the  slant  range  to  the  aircraft  when  a signal  (ESTIMATE  NOW)  was  given 

by  the  instructor.  These  estimates,  one  while  the  aircraft  approached  and 
one  after  the  aircraft  passed  over  the  site,  were  recorded  by  each  obser- 
ver on  special  record  forms.  Immediately  after  the  second  estimates  were 
made,  the  instructor  announced  the  correct  ranges  at  the  time  the  signals 
were  given.  At  that  time,  by  referring  to  his  record  form,  each  observer 
could  immediately  determine  his  error  of  estimation. 

b.  The  observers  had  been  told  specifically  that  they  were  going 
to  be  trained  to  accurately  estimate  four  different  ranges;  400,  800,  1,500, 
and  2,500  meters;  but  that  during  training,  the  signal  to  estimate  would  be 
when  the  aircraft  was  anywhere  from  300  meters  to  2,900  meters  from  them. 

c.  During  each  day's  training  session  the  aircraft  flew  36  passes, 
18  in  each  direction.  On  each  trial  (aircraft  pass)  two  estimates  were 
made,  one  incoming  and  one  outgoing.  Over  the  three  days  of  training,  each 
observer  made  a total  of  2l6  estimates. 
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d.  The  signals  to  estimate  given  by  the  instructor  were  based 
upon  a timing  system  which  relied  for  accuracy  upon  the  ability  of  the 
aircraft  to  maintain  a programmed  speed  and  course  within  some  limits. 

In  order  to  check  the  accuracy  of  this  system  of  determining  true  air- 
craft range , as  the  aircraft  passed  over  the  training  site  during  test 
trials,  a "crossover"  mark  was  put  onto  the  event  recorder  used  to  record 
observer  test  responses.  By  canparing  the  timing  system  with  these 
crossover  marks,  it  was  discovered  that  errors  existed  in  the  system. 

These  errors  resulted  in  training  the  observers  to  use  a different 
"yardstick"  them  was  programmed.  That  is,  when  the  observers  were  told 
during  training  that  the  aircraft  was  400  meters  away,  incoming,  it  was 
actually  closer  to  525  meters  away.  Table  Bl-1  shows  the  actual  ranges 
of  the  aircraft  when  the  observers  were  told  it  was  at  the  programmed 
range. 

4.  TEST  PROCEDURE. 

a.  A total  of  four  12-trial  tests  were  scheduled,  one  before 
training  commenced  on  the  first  test  day  and  one  at  the  end  of  each  of  the 
three  days  of  training. 

b.  Just  before  the  aircraft  began  a pass,  the  observers  were  told 
the  two  specific  ranges  they  were  to  estimate  on  that  pass,  one  incoming 
and  one  outgoing.  They  were  told  to  indicate  when  they  believed  that  the 
aircraft  was  at  the  specified  ranges.  Each  observer  was  provided  with  a 
pushbutton  connected  to  a channel  of  an  event  recorder.  When  the  observer 
through  the  aircraft  was  at  the  specified  ranges,  he  pressed  his  pushbutton. 

A mark  was  made  on  the  event  recorder  when  the  aircraft  was  at  the  programmed 
specific  ranges  for  the  purpose  of  checking  each  observer's  te6t  responses. 
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Course 


Assigned 

Range 


Actual 

Range 


Incoming 

400 

525 

800 

925 

. 1500 

1445 

2500 

2410 

Outgoing 

kOO  ; 

360 

800 

585 

1500 

1185 

2500 

2C40 

Table  Bl-1  Actual  Target  Ranges  During  Training 
Compared  to  Programmed  Ranges 
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c The  aircraft  scheduled  to  fly  the  test  trials  on  the  final  day 
had  mechanical  problems  and  did  not  fly.  As  a result , the  final  test 
trials  had  to  be  cancelled.  The  result  of  the  training  has  been  evaluated 
based  upon  performance  on  the  test  trials  at  the  end  of  the  second  day  of 
training. 

5.  RESUITS  OF  PRETRAINING  TEST. 

a.  The  l6  observers  that  were  trained  varied  in  age  f^am  .18  to 
24  years.  All  had  20/20  vision,  uncorrected  or  corrected.  GT  scores 
ranged  from  83  to  130,  with  a mean  of  111  and  standard  deviation  of  11. 

b.  At  the  beginning  of  the  first  day  of  training,  the  observers 
were  told  that  they  were  to  be  given  some  training  in  range  estimation  to 
aerial  targets,  but  that  first  they  would  be  tested  to  see  how  well  they 
could  estimate  various  ranges  before  training.  The  results  of  the  pre- 
training test  are  indicated  in  Table  Bl-2. 

c.  The  incoming  ranges  of  400  meters  and  800  meters  and  outgoing 
ranges  of  400  meters  were  greatly  overestimated.  This  shows  that  the 
observers  believed  that  these  ranges  were  much  greater  distances  than  they 
actually  were.  The  remaining  means  of  estimates  were  accurate,  but 
variation  was  relatively  large  in  all  cases. 


Course 


Estimated 

fiance 


Actual 

Range 


Standard 

Deviation 


Incoming 

400 

797 

162 

800 

1394 

259 

1500 

1752 

238 

2500 

2790 

282 

Outgoing 

4oo 

711 

154 

800 

764 

84 

1500 

1514 

123 

2500 

2317 

203 
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APPENDIX  G 

ARKANSAS  POSTTEST  INTERVIEWS 

1. (U)  OBJECTIVE. 

a.  A 6et  of  questions  was  asked  of  the  l6  military  personnel  who 
served  as  observers  for  the  human  factors  studies  conducted  in  Oklahoma/ 
Arkansas  as  part  of  JTF-2  Test  3 - 1/ 3 • 5 (NF) . The  questions  were  presented 
during  individual  interviews  conducted  during  the  last  three  days  of  the 
testing.  All  interviews  were  conducted  by  one  human  factors  scientist. 

b.  The  interviews  were  prefaced  with  the  following  introductory 
statement : 

"Now  that  we  are  nearing  the  end  of  our  tests  here  in 
Arkansas,  we  have  a number  of  questions  to  ask  you  concerning  your  test 
activities  during  the  past  few  weeks.  It  is  hoped  that  your  answers  to 
these  questions  will  help  us  clear  up  the  minor  confusions  and  uncertainties 
that  always  appear  after  field  testing  is  done." 

2 . ( C ) INTERVIEW  QUESTIONS.  The  below  12  questions  were  asked  of  each 
observer  participating  in  the  test.  For  each  question  the  responses  are 
indicated  in  summary  form. 


a.  (U)  Qustion  No.  1.  "We  understand  that  many  of  the  observers  have 
missed  detecting  sane  of  the  aircraft  for  one  reason  or  another.  About  what 
percentage  of  the  aircraft  do  you  think  you  missed?  Or,  in  other  words,  out 
of  every  ten  flights,  how  many  did  you  not  detect  until  the  aircraft  was  at 
crossover  or  behind  you?" 
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(l)  Summary  of  Answers. 


Once 

Once  or  twice 
Two  or  three 


(2)  Individual  Answers. 


Observer  ID  No. 

Answer 

63 

Less  than 

1 

43 

1 

'1 

1 

13 

2 or  3 

51 

1 

62 

1 or  2 

23 

2 or  3 

12 

Less  than 

1 

4l 

Less  than 

1 

21 

Less  than 

1 

22 

Less  than 

1 

53 

1 or  2 

31 

1 or  2 

11 

1 

4c 

1 

52 

1 

b.(0)  Question  No.  2 "Have  you  missed  a greater  percentage  of  one 
kind  of  aircraft  than  another?" 


I 

I 1 

li 
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(l)  Answers. 

(a)  Nine  observers  indicated  that  the  F-^C  was  easiest  to 
detect  because  it  emitted  a lot  of  exhaust  smoke.  Observers  detected  smoke 
before  seeing  the  aircraft's  form. 

(b)  Six  of  the  observers  stated  that  the  F-105D  was 
hardest  to  detect  since  it  gave  off  very  little  or  no  smoke.  One  observer 
thought  the  F-105D  also  tended  to  fly  higher  and  slower. 

(c)  Two  observers  thought  the  A-6A  was  hardest  because  it 
gave  very  little  smoke. 

(d)  Four  observers  thought  there  were  no  differences  be- 
tween the  ease  of  detecting  the  three  types  of  aircraft. 

(e)  Two  observers  believed  that  the  camouflaged  F-4Cs  were 
difficult  to  detect  when  the  aircraft  were  viewed  against  a terrain  back- 
ground. The  rest  of  the  observers  either  had  no  opinion  or  said  there  was 
no  difference. 

c . (u)Question  No.  3«  "Why  do  you  think  you  missed  seeing  the  aircraft?" 
(l)  Answers. 

(a)  Six  observers  said  the  aircraft  were  off  course  (missed 
target)  to  the  side  or  penetrated  from  the  from  the  wrong  direction. 

(b)  Eight  observers  stated  the  aircraft  were  obscured  by 
trees  because  they  were  off  course. 


(c)  Three  observers  stated  they  were  day-dreaming  or  in  a 


trance -like  state. 
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(2)  Several  miscellaneous  reasons  were  offered  Including 
the  following: 

(a)  Looked  like  a bird. 

(b)  Missed  seeing  it  when  it  first  came  over  the  hill 
and  lost  it  in  the  trees. 

(c)  Glare  and  spots  in  eyes. 

(d)  Looking  in  the  wrong  direction. 

d.(C)  Questions  No.  4 & 5-  "What  search  angle  do  you  usually  use  at 
this  target  site?  Show  me  what  landmarks  you  use.  What  search  angles  do 
you  use  at  the  other  target  sites?" 


(l)  Summary  of  Answers 


Search  Anglt 
54  degrees 
80  degrees 
92  degrees 
70  degrees 


(2)  Individual  Answers. 


Observer 

Search  Angle  At 
W1  W4  E3 

E4 

M 

22 

30 

75 

40 

48 

11 

65 

80 

80 

60 

71 

42 

-- 

— 

90 

-- 

-- 

52 

40 

20 

20 

40 

30 

21 

90 

65 

120 

50 

81 

4l 

-- 

-- 

90 

-- 

— 

12 

20 

90 

30 

120 

65 
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Observer 

Search  Angle  (in  degrees)  At 
W1  W4  E4 

M 

23 

6o 

90 

180 

75 

101 

62 

6o 

90 

135 

120 

101 

51 

45 

45 

45 

45 

45 

13 

20 

180 

180 

45 

106 

6i 

70 

45 

1B0 

100 

99 

43 

60 

150 

60 

60 

82 

63 

45 

45 

45 

45 

45 

53 

90 

90 

120 

120 

105 

31 

6o 

60 

60 

60 

60 

N 

l4 

14 

16 

14 

M 

54 

80 

92.5 

70 

2 

0 

454.21 

1658.80 

1781.63 

914.28 

e.(U)  Question  No.  6.  "Do  you  have  a favorite  location  at  each 
target  site,  and  do  you  always  go  to  the  same  location?" 

(l)  Answers.  All  observers  except  two  answered  affirmatively 
to  this  question.  There  were  two  "nonconformists"  in  one  group  that  rotated 
positions  (Numbers  12  and  13  claimed  to  have  been  at  all  ORB  locations  at 
all  four  target  sites,  but  this  assertion  was  not  supported  by  the  other  two 
men  in  that  group). 

f . (U)Question  No.  7.  "What  is  your  location  at  each  target  site?" 

(The  relative  observer  positions  at  each  site  are  shown  in 
Figures  C-l-1  through  Cl-4.  The  position  numbers  are 
indicated  in  brackets.) 
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(l)  Answers.  The  location  of  each  observer  at  each  target 
site  is  shown  below. 


Observer 

Location  at  Target  Site 

Number 

W1 

w4 

12 

E4 

11 

4 

4 

1-4 

1,3,4 

12 

1-4 

1-4 

1-4 

1-4 

13 

1-4 

1-4 

1-4 

1-4 

22 

4 

l 

l 

3 

21 

2 

2 

3 

4 

23 

1 

4 

2 

2 

31 

2 

3 

4 

1 

in 

2 

2 

3 

2 

42 

3 

4 

1 

4 

43 

4 

4 

4 

4 

51 

l 

l 

1 

3 

52 

4 

2 

1 

4 

53 

3 

l 

4 

1 

61 

4 

3 

2 

3 

62 

2 

4 

1 

2 

63  2 2 2 3 

g.  (U)Question  No.  8.  "Do  the  four  test  monitors  do  their  job  in  the 
same  way?  For  example,  do  all  the  monitors  give  you  the  same  amount  of 
early  warning?" 


(l)  Answers. 

(a)  Seven  observers  stated  all  four  monitors  did  their  Job 

in  the  same  way. 

(b)  Three  men  said  Monitor  A put  the  men  on  search  earlier 
for  the  no  early  warning  trials. 
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(c)  Three  other  men  said  Monitor  B put  the  men  on  Bearch 
earlier  for  the  no  early  warning  trials. 

(d)  Three  additional  men  said  both  Monitors  A & B put  the 
men  on  search  earlier. 

h.  (u)Question  No.  9.  "Have  you  picked  up  early  warning  information 
from  the  radio,  particularly  on  the  no  early  warning  trials?" 

(l)  Answers. 

(a)  All  of  the  men  answered  affirmatively  to  this  question. 

(b)  The  men  felt  the  radio  report  of  Zulu  Time  for  the 
observers  at  W1  was  particularly  helpful. 

(c)  When  the  observers  were  at  W4,  E3,  and  E4,  hearing  Echo 
or  Zulu  Time  did  not  aid  them  very  much  because  the  "time  over  target" 
varied  considerably,  and  there  were  many  times  the  aircraft  missed  these 
targets . 

i.  (U) Question  No.  10.  "How  many  days  passed  before  you  had  a pretty 
accurate  idea  where  the  aircraft  would  appear  at  each  target?" 

(l)  Answers. 

(a)  Ten  observers  Bald  by  the  end  of  the  first  test  week. 

(b)  Three  observers  weren't  sure  until  after  the  second  day 
they  were  stationed  at  each  target. 
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(c)  Two  men  said  they  were  sure  after  the  first  three 
flights  at  each  target  during  the  first  week. 

(d)  One  man  said  he  knew  by  the  second  test  day  because 
the  GSIP  antennas  were  centered  on  the  flight  path. 

J.(U)Que8tion  No.  11.  "If  you  were  the  NCOIG  of  a caliber  .50  MO, 
how  would  you  organize  your  crew  for  the  search  and  detection  functions 
required  for  defense  of  a iBO-degree  perimeter?" 

(l)  Answers.  The  l4  different  answers  are  listed  below. 

(a)  Two  men  rotating  an  duty  every  15  minutes. 

(b)  Four  men  on  duty  for  one-half  hour  watch. 

(c)  Three  men  on  watch  for  one  hour  if  it  is  cool  weather. 
If  It's  hot,  three  men  for  one -half  hour. 


this  method). 


(d)  Two  men  for  one  hour. 


(e)  Two  men  for  two  hours  (three  observers  mentioned 


(f)  Two  men  for  three  hours. 


(g)  Three  men  for  one  hour. 


(h)  Two  men  for  as  long  as  necessary,  each  with  a 90-degree 


sector. 


(i)  Two  men  for  four  hours,  but  have  them  trade  positions 


every  10  minutes. 
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(j)  Three  men  for  one  to  one  and  one -half  hours. 


(k)  Four  men  for  one  to  one  and  one-half  hours. 


(l)  Four  men  for  tvo  hours. 


(m)  Six  men  for  tvo  hours. 

(n)  Three  men  for  one  hour,  each  vith  a 60-degree  sector. 

k.  (U)Question  No.  12.  "This  next  question  concerns  range  estimation, 
and  consists  of  tvo  parts.  For  some  of  the  target  sites,  the  low-flying 
aircraft  have  a tree  or  terrain  background;  for  other  targets  the  aircraft 
are  seen  against  a sky  background.  Does  this  difference  in  backgrounds 
affect  your  ability  to  estimate  range?  What  techniques  do  you  use  for 
estimating  range?" 

(l)  Ansvers. 

(a)  The  ansvers  to  part  one  of  this  question  (sky  or 
terrain  easier)  vere  almost  equally  divided  among  three  alternatives; 

1 Sky  background  is  easier:  4 

2 Tree  background  is  easier:  5 


2,  No  difference: 


4 No  ansver: 


(b)  The  answers  to  part  two  (techniques  used  in  estimating 
range)  of  the  question  were  as  follows: 


Technique  Frequency  Observer  ID 

Apparent  size  if  aircraft 

is  high  and  aircraft  details  3 11,13,43 

if  it  is  low. 


Aircraft  size  only. 

Distance  to  terrain  if 
aircraft  is  low  and  size 
if  it  is  high. 

Distance  to  terrain. 

Aircraft  details. 

No  information  obtained 


1 12 

k 53,63,22,21 

1 4l 

3 62,42,61 

4 52,23,31,51 
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APFENDDC  D 

DATA  PROCESSING  AND  ANALYSIS 

by 

Michael  Carter 

Sandia  laboratories  ,, 

1.  GENERAL.  Th?  data  processing  consisted  of  reducing  and  editing  the 
Test  3-1/3. 5 (NF)  Oklahoma/Arkansas  data,  time  relating  this  event  data 
with  applicable  Test  4.1  aircraft  position  data,  and  creating  a data  bank 
for  use  in  the  analysis  effort. 

2.  QUALITY  CGNTROL  OPERATIONS. 

a.  Th*  ev^nt  data  were  recorded  on  magnetic  tape  in  the  orbiting 
C-130  aircraft.  These  tapes  were  processed  through  a playback  station 
and  formated  for  the  CDC-3400  computer.  The  event  data  were  processed  by 

a program  called  QUICKORB,  which  provided  preliminary  information  which  was 
transmitted  to  the  Mena  Test  Range  concerning  the  availability  of  event 
data  from  completed  trials. 

b.  The  data  event  tapes  were  subs  juently  edited  manually  to  remove 
invalid  rials.  An  observation  was  considered  invalid  for  any  of  the 
following  reasons: 

(1)  the  aircraft  had  not  passed  within  2000  meters  of  the  ground 

target, 

(2)  the  aircraft  was  grossly  off  course;  that  is,  it  penetrated  the 
target  area  outside  the  assigned  iSO-degree  search  sector,  and 

(3)  aircraft  position  data  were  not  available. 
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c.  Each  response  was  evaluated  to  accuse  chut  the  individual  responses 
used  in  the  analysis  were  plausible . The  - e data  checks  were  incorporated 
into  the  data  reduction  prcgra.Tu.ing  efic  ~ . Much  of  the  rationale  for 
these  data  checks  was  based  upor.  manual  review  of  the  data.  The  field  data 
contained  many  situations  where  responses  were  not  in  sequence  or  occurred 
at  unacceptable  tinges  (outrange  estimates  prior  no  crossover,  etc.).  The 
qualify  control  standards  adopted  were  based  upon  the  characteristics  of 
the  target  or  geometrical  considerations.  Below  is  a list  of  the  data 
checks  used  for  deleting  ir.di vic.ua!  invaiie  observations : 

(1)  detect  occurred  after  crossover, 

(2)  the  pilot  acquired  after  crossover, 

(3)  an  inrange  estimate  occurred  after  crossover, 

(4)  an  outrange  estimated  occurred  prior  to  crossover, 

(5)  a time  difference  bexween  pilot  acquire  and  observer  detect 
greater  than  200  seconds  indicated  an  invalid  detect  event  because  at  the 
test  speeds  flown,  the  aircrai't  was  not  available  for  200  seconds, 

(6)  the  detect  event  must  occur  before  the  inrange  event,  and 

(7)  the  inrange  event  must  occur  before  the  outrange  event. 

3.  DATA  REDUCTION. 

a.  The  data  were  subsequently  time  related  to  preliminary  Test  4.1 
data  by  use  of  a program  called  CRBDATA.  The  time-related  data  were 
sorted  on  magnetic  tape  for  use  in  the  final  processing. 
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b.  The  final  processing  consisted  of  time  relating  the  observer 
response  da-ca  to  the  final  Test  4.1  aircraft  position  data,  which  included 
computing  the  performance  measures  and  creating  the  data  bank  for  analysis. 

c.  The  ORBI&TA  program  time  related  Test  4.1  data  to  the  observer 
data,  computed  the  values  of  the  performance  measures  and  the  independent 
variables,  and  created  the  data  bank.  The  data  used  from  Test  4.1  were 
the  aircraft  position  and  the  pilot  acquire  events. 

(l)  The  terrain  data  for  targets  Wl,  W4,  E3,  and  E4  were  also 
obtained  frcm  Test  4.1.  The  horizon  features  for  each  target  were 
selected  and  recorded  on  magnetic  tape.  These  horizon  features  consisted 
of  the  angle  of  elevation  above  and  below  the  target's  plane;  that  is, 
the  plane  tangent  to  the  earth  at  the  local  target.  The  horizon  eleva- 
tion angles  were  given  as  a function  of  the  azimuth  angle,  — 90  degrees, 
relative  to  the  line  connecting  the  target  and  the  FRP. 

d.  The  following  quantitative  variables  were  calculated  for  each 
occurrence  of  an  observer  response: 

(1)  local  slant  range  of  aircraft  (SR), 

(2)  horizon  subtended  angle  (HSA), 

(3)  sun  angle  (SA), 

(4)  aircraft  subtended  angle  (ASA), 

(5)  aircraft  speed  (S), 


(6) 


apparent  contrast  of  the  aircraft  (C) , 


(7)  total  angular  velocity  (u), 


(8)  slant  range  rate  (R) , 


(9)  azimuth  rate  (©), 


(10)  elevation  rate  (0),  and 


(ll)  aircraft  altitude  (altitude  above  target  plane)  (A). 


e.  The  following  performance  measures  were  calculated: 


(l)  detection  range  (R), 


(2)  inrange  algebraic  error  (El), 
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(3)  percentage  error  inrange  (Pi), 


(4)  outrange  algebraic  error  (EO), 


■ j 
: i 


(5)  percentage /error  outrange  (PO),  and 


4 
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(6)  pilot  acquisition  minus  observer  detection  time  interval  (lit). 

f.  These  data  were  time  related  and  recorded  on  magnetic  tape  along 
with  the  observer  response  time,  answers  to  questions,  and  training  and 
aptitude  scores.  Each  response  was  in  a separate  data  record.  ID  informa- 
tion such  as  aircraft  type  (a/c),  target  number  (ST),  early  warning  condi- 
tion (EW),  observer  number  (OB),  and  all  visibility  data  were  included  in 
each  data  record. 
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a.  Statistical  Methods.  This  section  describes  the  statistical 
techniques  and  computer  programs  used  in  the  analysis. 


k.  AMLYSIS. 


(1)  Descriptive  statistics  used  included  histograms,  cumulative 
frequency  plots  and  scatter  diagrams  (S CD).  Included  with  the  histograms 

V 

were  the  mean,  standard  deviation,  median,  maximum  and  minimum  values  for 
the  data  used.  Cumulative  frequency  plots  were  derived  from  the  histograms. 
SCD  wex'e  merely  x - y plots  for  specified  pairs  of  continuous  variables, 

(2)  Analysis  of  varix":s  (AiJCVh),  analysis  of  covariance  (AKCOVA), 
and  regression  analysis  v ; the  techniques  employed  through  the  analysis 
phase.  These  analyses  were  accomplished  by  the  use  of  two  computer  pro- 
grams described  below. 

b.  Weighted  Regression  Analysis  Program  (WRAP).  WRAP  is  a computer 
program  that  performs  the  calculations  required  of  multiple  linear  regres- 
sion (see  reference  3)*  Observations  can  be  weighted  and  data  can  be 
transformed  in  the  program.  The  program  selects  a significant  subset  of 
independent  variables  by  a fixed  F value  or  fixed  probability  level.  Also 
included  is  a flexible  system  for  testing  hypotheses  In  balanced  or 
unbalanced  designs.  The  method  o.  analysis  is  the  usual  least  squares 
method  for  obtaining  estimates  of  the  regression  parameters.  The  output 
from  WRAP  normally  consisted  of  the  following: 

(1)  all  correlation  coefficients  (between  X.,  X.,  and  X.,  y), 

u C f 

(2)  (X'X)  (the  variance -covariance,  or  the  "C"  matrix  for  the 
regression  parameters). 


(3)  ANOVA  for  total  regression  model, 
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(4)  multiple  regression  coefficient  (b,.  's),  R,  R , residual 
variance  and  standard  deviation, 

(5)  each  regression  coefficient,  its  standard  error,  t value  F 
ratio,  and  sum  of  squares, 

(6)  residual  information  consisting  of  observed  y,  predicted  y, 
observed-predicted,  (observed-predicted)/residual  standard  deviation, 
weight  value  (=  one  unless  weights  " "c  assigned),  standard  deviation  of 
observed -predicted  value,  and 

(7)  AUOVA,  where  factor  sums  of  squares  are  adjusted  for  other 
factors  included  in  the  AWOVA. 

c.  Multivariate  Analysis  of  Variance  (MANOVA).  MANOVA  is  a computer 
program  that  performs  univariate  and  multivariate  analysis  of  variance, 
covariance  and  regression  (see  reference  4).  It  handles  balanced  and 
unbalanced  designs,  including  missing  cells.  The  outputs  usually  obtained 
in  this  analysis  are  as  follows: 

(1)  the  complete  ANOVA  table  containing  factors  sums  of  squares 
adjusted  for  other  factors  in  the  analysis, 

(2)  contrasts  for  factor  levels  and  interactions  in  the  model. 

With  respect  to  main  effects,  the  contrast  is  the  deviation  of  the  treat- 
ment mean  from  the  grand  mean.  In  the  case  of  two-way  interactions,  the 
contrast  is  the  value  of  y..  -y.  -y  . +y  , which  represents  the  deviation 

i j • iii  • j • ••• 

of  the  cell  mean  from  the  grand  mean  corrected  for  associated  main  effects, 

(3)  correlations  of  contrasts  with  standard  deviations  of  con- 
trasts, divided  by  standard  deviations  of  variables  on  the  diagonal,  is 
printed  out  in  matrix  form.  These  values  are  used  to  compute  the  variances 
of  contrasts  or  linear  combinations  of  contrasts. 
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(4)  Within  ce~i  coefficients  and  the  "C"  matrix  (X'X)  ^ (with 
sums  of  squares  regression  in  ANOVA  'cable)  when  covariables  are  present. 

d.  Both  the  MANCVA  and  'WRAP  programs  have  properties  useful  to  the 
analyst.  In  the  early  analysis  phase,,  WRAP  was  used  exclusively  due  to  the 
availability  of  correlations  between  the  variables,  residuals  and  indivi- 
dual sums  of  squares  for  all  variables.  In  the  final  analysis,  MANOVA  and 
WRAP  were  utilized  on  the  same  modal  as  each  program  contained  information 
only  available  from  the  other  through  much  hand  manipulation  of  the  data. 

5.  IMPACT  OF  DATA  REDUCTION  CN  THE  ANALYSIS  PLAN. 

a.  The  test  design  has  been  discussed  in  Section  1,  paragraph  Z,  and 
quality  control  operations  in  Section  3>  paragraph  2.  Problems  were  encoun- 
tered in  the  aircraft  position  data  obtained  by  Test  4.1  instrumentation  and 
responses  obtained  for  Test  3 - 1/3 • 5 (iff).  Data  losses,  absence  of  unmask 
data,  and  failure  to  fly  assigned  altitude  and  speed  made  it  necessary  to 
alter  the  original  analysis  plan.  Each  of  these  problems  is  discussed 
below. 


(1)  The  rationale  for  deleting  data  was  given  in  the  quality 
control  section.  Additional  losses  resulted  from  the  absence  of  Test  4.1 
position  data  and  pilot  behavior;  e.g.,  on  targets  W4  and  E4,  the  pilots 
frequently  switched  off  test  instrumentation  causing  substantial  losses  in 
outrange  estimation  data. 

(2)  The  unreliability  of  unmask  range  and  time  resulted  in  its 
deletion  as  a quantitative  variable.  This  caused  major  revisions  in  the 
detection  analysis.  The  analysis  was  redirected  to  investigating  the 
target  and  altitude  effects  in  the  hope  that  these  variables  would  be 
closely  related  to  unmask  range. 


! 
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(3)  The  pilots  1 failure  to  fly  assigned  speeds  and  altitudes 
resulted  in  a confounding  of  these  variables  with  other  test  factors.  As 
a result  of  this  confounding,  the  study  planned  for  comparing  the  effect 
of  three  speeds  (3^0  knots,  b20  knots  and  >50  knots)  for  the  F-4C  at  alti- 
tudes of  0 to  500  feet  was  no  longer  possible.  An  attempt  was  then  made 
to  evaluate  detection  performance  for  the  F-kC  using  speed  and  altitude  as 
quantitative  variables. 

6.  RATIONALE  FOR  THE  ANALYSIS  PIAN.  This  paragraph  defines  the  basic 
analysis  flow  that  was  originally  planned.  Subsequent  paragraphs  will 
elaborate  on  how  the  analysis  progressed  in  actuality. 

a.  Initially,  the  various  plots  were  to  be  reviewed  and  possible 
areas  of  investigation  were  to  be  noted.  SOD  would  be  the  major  source 
of  information  concerning  quantitative  variable -response  relationships, 
while  histograms,  means  and  standard  deviation  would  give  indications  of 
the  effects  of  qualitative  variables. 

b.  Based  upon  the  observed  relationships  noted  in  the  SCD  and  histo- 
grams, a beginning  model  would  be  created  and  processed  using  WRAP.  Sub- 
sequent analyses  would  be  directed  towards  refining  the  significant 
relationship.  ANCOVA  models  would  be  used  in  this  phase.  WRAP  was  the 
program  to  be  used  as  the  correlations  and  individual  variables  informa- 
tion in  the  output  would  make  it  possible  to  3ee  improvement  quite  easily. 
Reduction  of  the  residual  error  would  be  the  primary  criterion. 

c.  As  the  final  series  of  models  were  selected,  the  models  would  be 
processed  using  both  WRAP  and  MAN0VA. 

d.  Models  would  be  formed  to  identify  significant  sources  of  variation 
in  the  data  with  predictive  capabilities  as  a secondary  consideration 
(although  WRAP  always  yields  a predictive  regression  equation).  This  would 
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result  in  considerable  utilization  of  qualitative  variables  unique  to  this 
test,  such  as  aircraft  type,  target,  or  early  warning,  that  would  not  be 
desirable  in  predictive  models.  The  quantitative  variables  were  to  be  used 
as  concomitant  information. 

7.  MODIFIED  DETECTION  ANALYSIS. 

a.  Three  sets  of  covariables  were  used  in  the  analysis  of  detection 
for  describing  aircraft  performance.  These  were: 

(1)  altitude,  speed,  minimum  range,  ASA,  HSA,  SA, 

• « • 

(2)  G,  0,  R,  ASA,  HSA,  SA,  and 

(3)  u>,  R,  ASA,  KSA,  SA. 

b.  The  qualitative  variables  included  in  the  analysis  at  various  times 
were  aircraft  type  (a/c),  target  (ST),  early  warning  (EW),  and  observer  (03) . 
The  descriptive  statistics  on  detection  range  utilized  these  variables. 
Descriptive  statistics  obtained  were  as  follows: 

(1)  histograms  and  cumulative  frequency  plots  on  detection  range 

for  each  classification  of  aircraft  type,  site,  observer,  early  warning, 

A/C  by  ST,  A/C  by  OB,  ST  by  OB,  and  all  data, 

♦ 

• • • 

(2)  SCD  across  all  trials  for  detection  range  versus  0,  0,  R,  ASA, 
HSA,  SA,  S,  A,  w,  CA,  Vis,  and  Rq,  and 

(3)  histograms  on  S and  A (computed  at  detection)  for  each  air- 
craft type,  site,  observer,  early  warning,  and  across  all  trials. 

c.  Based  on  information  presented  by  the  SCD  and  histograms,  a beginning 
function  was  evolved  for  each  covariable.  These  were  linear,  quadratic, 


D-9 


exponential  functions  that  had  simple  interpretations.  Some  beginning 

p • ■ ^ 

terms  were:  l/ASA,  EX?{  - (HSA)  /iOO},  SA,  l/f),  R,  SA,  A , S,  R , 

EX?{  - (©  - .002)  x 103},  and  £X?{  - (u  - .002)  x 103} . The  constants  in 
the  HSA,  © and  w expressions  were  estimates.  Throughout  the  analysis, 
the  functional  or  structural  relationship  was  most  important,  whereas  the 
constants  would  likely  be  unique  with  this  test. 

d.  The  WRAP  program  was  used  to  perform  regression  analysis  using 
covariables  from  each  system  and  the  qualitative  variables.  Correlations, 
residuals,  and  information  on  each  regression  coefficient  were  examined. 
Three  covariables  were  deemed  worthy  of  further  consideration  because  of 
their  relationships  with  detection  range.  These  were  l/ASA  (correlation 

= .9719)i  altitude  (correlation  <=  .6l 64),  and  speed  (correlation  = .2415). 
The  functions  involving  © and  w were  strongly  correlated  with  detection 
range,  but  were  nearly  perfectly  correlated  with  l/ASA,  whereas  l/ASA,  A, 
and  S were  poorly  correlated  with  each  ocher. 

e.  Many  quantitative  variables  were  not  useable  due  to  the  complexity 
of  the  functional  relationships  between  them  and  the  dependent  variable. 

It  was  important  that  any  terms  used  as  covariables  must  be  of  interpre- 
table value.  These  decisions  to  remove  certain  variables  from  consideration 
were  based  mostly  on  WRAP  results,  since  the  scatter  plots  could  not  be  used 
to  evaluate  the  influence  of  the  covariable  in  the  presence  of  qualitative 
variables. 

f.  The  next  step  was  to  conduct  a series  of  straight  AjNOVA  to  assess 
the  effects  of  the  qualitative  variables  in  the  absen:e  of  any  covariables. 
This  was  done  via  WRAP  since  individual  factor  levels  could  be  investigated 
in  addition  to  the  usual  AITOVA  evaluation.  Prior  to  the  AITOVA  there  was 

no  indication  that  problems  were  present  in  the  data.  Considerable  care 
had  to  be  exercised  in  incorporating  the  covariables  into  the  ANCOVA  models. 
In  order  to  determine  if  there  was  confounding  between  the  qualitative  and 
quantitative  variables  (other  than  by  using  a long,  exacting  computational 
cycle),  it  was  necessary  to  examine  the  effects  of  qualitative  variables  with 
and  without  covariables  present. 
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The  AHOVA  gave  the  first  indications  of  problems  in  the  data.  Prior 

to  any  analysis  it  was  known  that  the  design  was  very  unbalanced.  Table  3-1 

presents  the  data  matrix  showing  the  number  of  observations  which  were 

obtained  for  each  combination  of  observer,  target,  aircraft,  and  early 

warning  level.  The  lack  of  observations  in  many  of  the  cells  of  Table  3-1 

**  - 

indicates  the  degree  of  unbalance.  3y  running  models  using  main  effects 
only  and  later  including  interaction  terms  that  were  estimable,  one  can 
observe  how  ohs  inbalance  and  confounding  affected  the  results.  This 
information  was  available  when  the  AKCCVA  were  conducted. 

g.  The  variable  aircraft  subtended  angle  (ASA)  was  deemed  unsuitable 
for  future  use  as  a covariable  as  it  was  essentially  another  measure  of 
range.  However,  ASA  is  normally  not  available  in  detection  studies  and 
some  utilization  was  desirable.  Thus,  the  following  information  was  pro- 
vided on  ASA  and  detection  range: 

(1)  SCO  of  detection  range  versus  ASA  for  all  trials,  each  aircraft 
type,  each  site,  each  warning  condition,  and  each  observer, 

(2)  SOD  of  cumulative  percent  detection  versus  ASA  and  l/ASA  for 
all  trials,  each  aircraft  type,  each  site,  each  warning  condition,  and 
each  observer, 

(3)  for  each  classification  in  (2),  the  following  regression 
equations  were  provided; 

(a)  C?  = 1 - A * EXP{  B(Detection)} , 

(b)  CP  = A + B Ln  R, 

(c)  CP  = A + B Ln  ASA, 
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CP  = 1 - A • EX?{  B(A£A); , 


* 


(<0 


(e)  Detection  Range  - 1 - A • EX?{  B(ASA)} , and. 


(f)  Detection  Range  - k/ (ADA;  . 

h.  Speed  (s)  and  altitude  (A)  were  included  individually  and  together 
in  ANCOVA  models.  Types  of  models  used  were  as  follows: 

(1)  yiJkl  = V + A/C.  + STj  * EWk  + (A/C  x ST)i(j  + (ST  x EW  Jk  + 
0xAijkl  + ^ijkl  + ®ijkl' 


(2)  y.Jkl  = p - A/Ci  + STj  + EWk  + (a/C  x ST)^  + (ST  x EW)JR  + 
6lSijkl  "^Jkl  + eijkl'  and 


(3)  y 


ijkl 


3lAijkl  + ^ijki 


= p + A/C.  + ST.  + EW.  + (A/C  x ST) + (ST  x EW).  + 

1 J K lj  j K 

+ e3Sijkl  + Vijkl  + VA'S)ijkl  + eijkl- 


Most  combinations  were  tried  with  the  qualitative  variables  (aircraft; 
ST;  EW)  being  the  only  termq  common  to  the  series  of  models.  Both  WRAP 
and  MANOVA  were  used  to  ana  :/ze  the  models.  By  reviewing  the  contrasts 
in  MATJOVA  and  individual  regression  coefficient  information  in  WRAP,  it 
was  possible  to  identify  rr.  Any  sources  of  confounding. 


i.  Variability  in  aircraft  altitudes  measures  at  the  various  event 
times  caused  many  of  the  problems.  A correlation  of  .62  was  obtained 
between  aircraft  altitude  above  target  plane  and  detection  range  for  all 
trials.  Correlations  ranging  from  .33  to  -90  were  obtained  for  the  F-uC 
aircraft  alone  when  computed  separately  for  each  of  the  four  ground  tar- 
gets. However,  the  magnitude  of  these  corrections  is  misleading.  For 
all  targets,  the  ground  level  tended  to  slope  upwards  from  the  target 
along  the  expected  flight  path.  Since  the  aircraft  tended  to  fly  at 
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approxima  the  same  altitude  above  the  local  ground  level,  their 
altitude  abv^ve  the  target  plane  gradually  decreased  as  they  approached 
the  target.  Therefore,  aircraft  detected  at  greater  ranges  would  be 
expected  to  be  at  higher  altitudes  than  those  detected  at  closer  ranges. 
Furthermore,  the  targets  were  effected  differently  as  the  angle  of  the 
slope  varied  from  target  to  target.  For  this  reason,  it  was  impossible 
to  evaluate  the  effect  of  altitude  on  detection  range. 

j.  The  above  is  one  example  of  the  types  of  confounding  present. 
Altitude  was  cai founded  with  ST,  aircraft  and  EW.  The  tremendous  unbal- 
ance contributed  to  the  problem  and  definitely  prohibited  any  attempts 
to  solve  the  problem.  The  use  of  speed  encountered  problems  similar  in 
nature  to  those  of  altitude. 

k.  As  a result  of  the  findings  above,  the  emphasis  returned  to  ANOVA 

models.  To  remove  some  problems  and  get  a design  with  filled  cells,  a 
specific  sort  using  F-4C  aircraft  and  11  of  1 6 observers  was  analyzed. 
This  model  was  : y ^ + EW^  + OB^  + (ST  x EW).j  + (ST  x QB)ik 

+ (EW  x = eijkl-  Target  by  observer  interaction  was  the  most 

significant  factor,  and  this  prompted  some  further  investigations  of  this 
model  using  A and  S as  covariables.  However,  this  approach  did  not  remove 
the  confounding  described  earlier.  An  additional  model  incorporating  ST 
by  EW  by  OB  was  analyzed.  The  significant  factors  were  ST,  ST  by  OB,  and 
ST  by  EW  by  OB  which  essentially  says  that  observers  behaved  differently 
at  different  sites  under  different  early  warning  conditions.  The  mean 
values  responsible  for  the  interactions  did  not  follow  any  consistent 
pattern. 


1.  The  final  step  was  to  investigate  the  observer  within  group  and 
between  group  behavior.  This  was  as  a result  of  the  ST  by  EW  by  OB  inter- 
action. This  led  to  the  final  decision  that  the  variables  employed  did 
not  explain  the  situation.  The  responses  were  apparently  dependent  o' 


the  somewhat  unique  aircraft  trajectory  of  each  trial  and  the  target 
terrain.  No  suitable  variables  were  available  to  sufficiently  describe 
the  data.  '* 

m.  In  summary,  the  extensive  analyses  performed  on  the  detection 
data  yielded  little  in  the  way  of  positive  results.  The  reasons  above 
are  sufficient  to  preclude  any  meaningful  findings.  Statistical  analyses 
yielded  many  relationships  that  seemed  to  demonstrate  the  basic  problems 
in  the  data.  Any  analysis  more  sophisticated  than  descriptive  statistics 
was  not  effective  due  to  the  imbalance  and  confounding  present. 

8.  STUDIES  OF  PILOT  ACQUISITION  MENUS  OBSERVER  DETECTION  TE-2  INTERVAL  (j^) • 

a.  This  performance  measure  was  not  analyzed  extensively.  The  pre- 
liminary analyses  yielded  findings  similar  to  those  on  detection  range. 

The  plots  on  ^ utilized  in  the  analyses  were  as  follows: 

(1)  histograms  and  cumulative  frequency  plots  on  ^ for  each 
classification  of  aircraft  type,  site,  observer,  early  warning,  A/C  by  ST, 

AC  by  OS,  ST  by  OB,  and  all  data,  and 

• • • 

(2)  SOD  across  all  trials  for  versus  0,  0,  R,  SR,  ASA,  HSA,  SA, 

u 

S,  A,  u)  and  C^  computed  at  detection  and  pilot  acquire.  Additional  SCD  on 
were  on  S,  A,  ©,  0,  R,  ASA,  SA,  C. , w and  SR  at  detection  versus  the  same 
variable  computed  at  pilot  acquire. 

b.  In  addition  to  computation  of  descriptive  statistics,  several  WRAP 
were  conducted.  Inspection  of  these  analyses  did  not  reveal  any  meaningful 
relationships  between  and  the  quantitative  and  qualitative  variables. 
Because  of  the  concomitant  difficul~ies  with  evaluation  of  tne  detection 
data,  further  analysis  of  ^ was  not  made. 
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c.  Useful  information  or.  ole  range  of  differences  between  pilot  and 
observer  performance  is  contained  in  no  inferences  on  why  varied 

are  possible  with  the  data  presently  available  because  of  the  somewhat 
unique  trial  trajectories  which  affected  both  the  acquire  and  detection 
events. 

9.  RANGE  ESTIMATION  STUDIES . 

a.  The  problems  associated  with  range  estimation  (HE)  were  slightly 
different  than  those  associated  with  the  detection  studies.  Angular 
rates,  altitude,  speed,  minimum  crossing  range  (Rq) , detection  range  ar.d 
ASA  were  the  only  variables  computed  that  would  seemingly  effect  distance 
estimation. 

b.  The  procedure  used  for  analyzing  inbound  and  outbound  estimates 
followed  the  procedure  used  on  detection  data.  Many  of  the  unusual  rela- 
tions discovered  in  the  direction  data  carried  over  to  the  RE.  The  set  of 
qualitative  variables  differed  as  EW  was  not  used  and  assigned  ranges  were 
included  (used  as  a qualitative  variable  or  a quantitative  variable).  The 
descriptive  statistics  utilized  in  the  analysis  were  as  follows: 

(1)  histograms  and  cumulative  frequency  plots  on  El  and  EQ  for 
each  classification  of  aircraft  type,  site,  observer,  assigned  range, 

A/C  by  ST,  A/C  by  GB,  ST  by  03,  A/C  by  assigned  range,  OB  by  assigned  range, 
ST  by  assigned  range,  F-4C  A/C  by  ST  by  assigned  range,  and  all  data, 

• • • 

(2)  SCD  on  El  and  SO  versus  G,  0,  R,  ASA,  HSA,  SA,  S,  A,  and 
computed  at  inrange  and  outrange  estimates,  respectively.  Additional 

SCD  on  El  and  E0  versus  Rq,  assigned  range  and  RE  training  score  for  inrange 
and  outrange  estimates,  respectively,  were  plotted. 
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c.  For  the  analysis  of  El,  the  quantitative  variables  retained  for 
extensive  investigations  were;  inrange  training  score,  l/ASA,  l/u\,  9, 
i/ASA  x 'Juh,  outrange  training  score,  and  assigned  inrange  (AIR).  For 
the  analysis  of  EQ,  the  quantitative  variables  were;  minimum  crossing 
range,  l/ASA,  l/uq,  l/AGA  x i/u^,  outrange  training  score,  and  assigned 
outrange  (AOR) . The  interpretation  oi*  products;  i.e.,  l/ASA  by  l/wJj, 
were  discussed  and  valid  interpretations  were  considered  possible. 

d.  The  evaluation  of  AliGVA  yielded  additional  information  concerning 
the  confounding.  Less  data  was  available  resulting  in  greater  unbalance. 

e.  The  following  type  of  model  was  used  in  analysis  of  the  range 

estimation  data:  y^^  = t T A/C  = STj  + AjR^  + (a/C  x ST)^  + (a/c  x 
AjR)ik  + (ST  x AjR)jk  h-  &1(l/ASA)iJkl  - *2(l, M)ijkl  + Covariables 

were  tried  in  several  combinations,  and  a series  of  models  resulted.  They 
were  processed  using  both  WRAP  and  ISAKOVA. 

f.  In  the  analysis  of  El,  the  qualitative  variables  affected  by 
covariables  were  aircraft,  AIR,  and  ST  by  AIR.  ST,  aircraft  by  ST  and 
aircraft  by  AIR  were  always  significant.  By  looking  at  contrasts  and 
regression  coefficient  data,  it  was  noted  that  l/ASA  and  l/uh  caused  the 
greatest  reversals.  Further  investigation  showed  these  variables  strongly 
correlated  ( *\/.50)  with  AIR.  This  resulted  in  evaluation  using  AIR  or 
l/ASA  and  l/w^  separately.  These  results  indicated  that  AIR  did  not 
behave  consistently.  The  ANOVA  showed  that  aircraft  by  AIR,  ST  by  AIR, 

ST  by  aircraft,  and  ST  were  significant.  The  results  were  not  meaningful; 
i.e.,  the  information  available  about  the  sites  gave  the  analysts  no 
insight  into  why  ST  by  AIR  should  be  significant. 
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g.  The  following  types  of  models  were  used  in  the  analysis  of  the 
range  estimation  data  employing  covuriables: 

W ■ » * A/°i  * STj  * <A/C  A 3->io  * * eijk- 

(2)  yiJk  - „ ’ ~/C  - SSj  - (A/0  * SI)y  - ^(1/**)^  ♦ 
S2(l/“yijk  * eijk'  ar-d 

(3)  yir  = U ♦ A/C  - ST  - (A/C  x ST).,  - fi- (1/ASA)  + 

eaW-^Uk  * s3(i/asa  * i/JOy*  - 

h.  A similar  set  of  models  using  6 in  place  of  l/urs  was  analyzed. 

The  results  showed  reversals  due  to  the  ccvariable  present.  ST  was  the 
only  qualitative  variable  consistently  significant;  whereas,  aircraft  was 
never  significant.  Interaction  of  aircraft  by  ST  was  very  sensitive  to 
the  covariable (s)  present. 
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carch  performed  by  KurRRO  Division  No.  5,  El  Paso,  Texas  (now  HumRRO 
terr.  Division,  El  Paso  Office)  under  Work  Unit  TESTAID. 
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iir  defense  systems 
aircraft  detection 
/■server  perfsrr.ar.ee 
ar.ge  estimation 


taring  a series  of  tests  to  provide  operational  and  technical  data  for 
evaluating  the  capabilities  o:  surface-based  air  defense  system?  against 
attacks  by  low- altitude  higk- performance  aircraft,  the  capabilities  of 
ground  observers  to  detect  a:.d  estimate  the  range  to  aircraft  flying  attack 
missions  against  prebriefed  ground  targets  were  examined.  Observer 
p erformance  was  obtained  for  three  aircraft  flying  three  programmed  speeds 

(CcKtiKuedpi 
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. ,nd  two  pro.-;ra.'.ed  altitudes.  Sixte.:i  observe r.s  searched  a it©*  udor  for 
?ach  trial,  with  early  v.amir.s  of  an  sireraft  approach  being  provided  fur 
s j;:?  trials.  Three  real  time  events --time  at  uetectio.i,  ti..e  .\h-w.  tn© 
aircraft  was  at  the  estimated  inbound  range,  and  t:nc  w:\on  the  aircraft  was 
at  the  estimated  outbound  ranj>e--were  recorded  for  each  observer  for 
each  trial . 
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